Implement Virtual FileSystem abstraction over ManagedCode.Storage #107

Copilot · 2025-09-01T20:36:58Z

This PR implements a comprehensive Virtual FileSystem (VFS) that provides filesystem-like operations over any ManagedCode.Storage blob provider (Azure, AWS, GCP, FileSystem). The VFS addresses the need for a unified, high-level interface for file and directory operations across different cloud storage providers.

Key Features

Filesystem Abstraction

File and directory operations with familiar filesystem semantics
Path normalization supporting both Windows and Unix path conventions
Efficient directory listing with pagination for large datasets
Virtual directory support (no physical directory objects required)

Performance Optimizations

ValueTask for cache-friendly operations (file existence checks, metadata access)
Aggressive metadata caching with configurable TTL
Streaming operations for memory-efficient large file handling
Chunked transfers with configurable buffer sizes

Concurrency & Safety

ETag-based optimistic concurrency control
Safe concurrent file modifications with conflict detection
Atomic operations where supported by underlying storage

Advanced Capabilities

Multi-mount point support for accessing multiple storage providers
Progress reporting for long-running copy operations
Native blob metadata usage (no extra files created)
Custom metadata support with provider-specific prefixes

Usage Example

// Setup with dependency injection
services.AddVirtualFileSystem(storage, options => {
    options.DefaultContainer = "documents";
    options.EnableCache = true;
    options.CacheTTL = TimeSpan.FromMinutes(5);
});

var vfs = serviceProvider.GetService<IVirtualFileSystem>();

// File operations
var file = await vfs.GetFileAsync("/reports/2024/quarterly.pdf");
await file.WriteAllBytesAsync(pdfData);

// Directory operations with pagination
var reports = await vfs.GetDirectoryAsync("/reports");
await foreach (var entry in reports.GetEntriesAsync(pageSize: 50))
{
    Console.WriteLine($"{entry.Type}: {entry.Name}");
}

// Copy with progress tracking
await vfs.CopyAsync("/source/data", "/backup/data", 
    new CopyOptions { Recursive = true },
    progress: new Progress<CopyProgress>(p => 
        Console.WriteLine($"{p.PercentageComplete:F1}% complete")));

Architecture

The VFS is built on top of the existing ManagedCode.Storage layer and maintains full compatibility with all storage providers. It uses native blob metadata for storing VFS-specific information without creating additional files, ensuring optimal performance and storage efficiency.

The implementation follows the established patterns in the codebase and integrates seamlessly with the existing dependency injection and configuration systems.

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot · 2025-09-01T20:36:59Z

@KSemenenko 👋 This repository doesn't have Copilot instructions. With Copilot instructions, I can understand the repository better, work faster and produce higher quality PRs.

I can generate a .github/copilot-instructions.md file for you automatically. Click here to open a pre-filled issue and assign it to me. I'll write the instructions, and then tag you for review.

KSemenenko · 2025-09-01T20:38:22Z

@copilot use vnext branch as a base

KSemenenko · 2025-09-01T20:40:38Z

@copilot install dotnet 9 , use vnext brash as base, and write file with instruction for copilot

Co-authored-by: KSemenenko <[email protected]>

Copilot · 2025-09-01T20:45:18Z

@copilot install dotnet 9 , use vnext brash as base, and write file with instruction for copilot

✅ Completed all requested tasks:

Installed .NET 9: Successfully installed .NET 9.0.304 SDK and verified the project builds correctly
Switched to vNext branch: Now using vNext as the base branch
Created Copilot instructions: Added comprehensive .github/copilot-instructions.md with project overview, development guidelines, and technical context

The environment is now configured with .NET 9 and includes detailed instructions to help Copilot provide better assistance. Commit: 93223f4

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

aka.ms
- Triggering command: curl -I -sSL --retry 5 --retry-delay 2 --connect-timeout 15 REDACTED (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

KSemenenko · 2025-09-01T21:16:53Z

@copilot now implement this issue #106

Co-authored-by: KSemenenko <[email protected]>

Copilot · 2025-09-01T21:34:01Z

@copilot now implement this issue #106

I've successfully implemented the Virtual FileSystem as requested in issue #106. The implementation includes:

Core Features:

Complete VFS abstraction over blob storage (Azure, AWS, GCP, FileSystem)
File and directory operations with efficient pagination
Path normalization and validation
ETag-based concurrency control
Metadata management using native blob metadata
Streaming operations for large files
Multi-mount point support

Key Components:

IVirtualFileSystem - Main VFS interface
IVirtualFile / IVirtualDirectory - File/directory operations
VfsPath - Robust path handling
VfsWriteStream - Memory-efficient write streaming
Service registration extensions for DI

The solution now builds successfully with .NET 9 and integrates seamlessly with the existing ManagedCode.Storage architecture. Ready for testing and usage!

Commit: ab47a4d

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

aka.ms
- Triggering command: curl -I -sSL --retry 5 --retry-delay 2 --connect-timeout 15 REDACTED (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Copilot

Pull Request Overview

This PR implements a comprehensive Virtual FileSystem (VFS) abstraction layer over ManagedCode.Storage blob providers, enabling filesystem-like operations across different cloud storage providers (Azure, AWS, GCP, FileSystem). The VFS provides a unified interface for file and directory operations with filesystem semantics, performance optimizations including caching and streaming, and advanced features like progress reporting and multi-mount support.

Key Changes

Added VFS core interfaces and implementations with filesystem abstraction over blob storage
Implemented metadata management system for storing VFS-specific information in blob metadata
Added comprehensive options system for configuring VFS behavior including caching, directory strategies, and concurrency control

Reviewed Changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
ManagedCode.Storage.sln	Added new VFS project and additional build configurations for x64/x86 platforms
ManagedCode.Storage.VirtualFileSystem/ManagedCode.Storage.VirtualFileSystem.csproj	Project file defining VFS package with .NET 9.0 target and required dependencies
ManagedCode.Storage.VirtualFileSystem/Core/*.cs	Core interfaces defining VFS abstractions (IVirtualFileSystem, IVirtualFile, IVirtualDirectory, etc.)
ManagedCode.Storage.VirtualFileSystem/Options/VfsOptions.cs	Configuration classes for VFS behavior including caching, directory strategies, and operation options
ManagedCode.Storage.VirtualFileSystem/Implementations/*.cs	Concrete implementations of VFS interfaces with caching, metadata management, and blob storage integration
ManagedCode.Storage.VirtualFileSystem/Metadata/IMetadataManager.cs	Metadata management system for storing VFS information in blob metadata with provider abstraction
ManagedCode.Storage.VirtualFileSystem/Streaming/VfsWriteStream.cs	Write stream implementation that buffers data and uploads on dispose for VFS file operations
ManagedCode.Storage.VirtualFileSystem/Exceptions/VfsExceptions.cs	VFS-specific exception types for error handling including concurrency, not found, and operation exceptions
ManagedCode.Storage.VirtualFileSystem/Extensions/ServiceCollectionExtensions.cs	Dependency injection extensions for registering VFS services with default metadata manager implementation
.github/copilot-instructions.md	Development guidelines and context for the ManagedCode.Storage project

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-02T05:56:50Z

ManagedCode.Storage.VirtualFileSystem/Streaming/VfsWriteStream.cs

+            try
+            {
+                // Upload the buffered data
+                UploadBufferedDataAsync().GetAwaiter().GetResult();


Calling GetAwaiter().GetResult() in a synchronous dispose method can cause deadlocks. Consider using ConfigureAwait(false) or implementing only the async dispose pattern for this scenario.

Copilot · 2025-09-02T05:56:51Z

ManagedCode.Storage.VirtualFileSystem/Streaming/VfsWriteStream.cs

+            catch (Exception ex)
+            {
+                _logger.LogError(ex, "Error uploading data during stream dispose: {BlobKey}", _blobKey);
+                throw;


Rethrowing the exception in DisposeAsync after logging it may not be appropriate. Consider swallowing the exception or providing a different error handling strategy since dispose operations typically should not throw.

Suggested change

throw;

// Swallow the exception to avoid throwing from DisposeAsync

Copilot · 2025-09-02T05:56:51Z

ManagedCode.Storage.VirtualFileSystem/Options/VfsOptions.cs

+            if (patternIndex < pattern.Length && (pattern[patternIndex] == '?' || 
+                string.Equals(pattern[patternIndex].ToString(), input[inputIndex].ToString(), comparison)))


Converting individual characters to strings for comparison is inefficient. Use direct character comparison with case sensitivity handling instead: char.ToLowerInvariant(pattern[patternIndex]) == char.ToLowerInvariant(input[inputIndex]) for case-insensitive comparison.

Suggested change

if (patternIndex < pattern.Length && (pattern[patternIndex] == '?' ||

string.Equals(pattern[patternIndex].ToString(), input[inputIndex].ToString(), comparison)))

if (patternIndex < pattern.Length && (

pattern[patternIndex] == '?' ||

(comparison == StringComparison.OrdinalIgnoreCase

? char.ToLowerInvariant(pattern[patternIndex]) == char.ToLowerInvariant(input[inputIndex])

: pattern[patternIndex] == input[inputIndex])))

ManagedCode.Storage.VirtualFileSystem/Implementations/VirtualFileSystem.cs

Copilot · 2025-09-02T05:56:52Z

ManagedCode.Storage.VirtualFileSystem/Core/VfsPath.cs

+            }
+        }
+
+        return "/" + string.Join("/", segments);


When all segments are removed (e.g., path with only '.' and '..' components), this will return '/' which is correct for root. However, if segments is empty but the original path was not root, this could be problematic. Consider adding a check for this edge case.

…ileSystem.cs Co-authored-by: Copilot <[email protected]>

Initial plan

df5ac47

Copilot AI assigned Copilot and KSemenenko Sep 1, 2025

Copilot started work on behalf of KSemenenko September 1, 2025 20:37 View session

Copilot stopped work on behalf of KSemenenko due to an error September 1, 2025 20:39
Copilot has encountered an error. See logs for additional details.

Copilot AI requested a review from KSemenenko September 1, 2025 20:39

Copilot started work on behalf of KSemenenko September 1, 2025 20:39 View session

Copilot stopped work on behalf of KSemenenko due to an error September 1, 2025 20:40
Copilot has encountered an error. See logs for additional details.

Copilot started work on behalf of KSemenenko September 1, 2025 20:40 View session

Add Copilot instructions and configure .NET 9 environment

93223f4

Co-authored-by: KSemenenko <[email protected]>

Copilot AI changed the title ~~[WIP] Virtual FileSystem~~ Setup .NET 9 environment and add Copilot instructions Sep 1, 2025

Copilot finished work on behalf of KSemenenko September 1, 2025 20:46

Copilot started work on behalf of KSemenenko September 1, 2025 21:17 View session

Implement core Virtual FileSystem functionality

ab47a4d

Co-authored-by: KSemenenko <[email protected]>

Copilot AI changed the title ~~Setup .NET 9 environment and add Copilot instructions~~ Implement Virtual FileSystem abstraction over ManagedCode.Storage Sep 1, 2025

Copilot finished work on behalf of KSemenenko September 1, 2025 21:34

KSemenenko marked this pull request as ready for review September 2, 2025 05:54

Copilot AI review requested due to automatic review settings September 2, 2025 05:54

Copilot AI reviewed Sep 2, 2025

View reviewed changes

Update ManagedCode.Storage.VirtualFileSystem/Implementations/VirtualF…

1dc5bf4

…ileSystem.cs Co-authored-by: Copilot <[email protected]>

KSemenenko closed this Sep 19, 2025

KSemenenko deleted the copilot/fix-106 branch September 19, 2025 20:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement Virtual FileSystem abstraction over ManagedCode.Storage #107

Implement Virtual FileSystem abstraction over ManagedCode.Storage #107

Uh oh!

Copilot AI commented Sep 1, 2025 •

edited

Loading

Uh oh!

Copilot AI commented Sep 1, 2025

Uh oh!

KSemenenko commented Sep 1, 2025

Uh oh!

KSemenenko commented Sep 1, 2025

Uh oh!

Copilot AI commented Sep 1, 2025 •

edited

Loading

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

KSemenenko commented Sep 1, 2025

Uh oh!

Copilot AI commented Sep 1, 2025 •

edited

Loading

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Sep 2, 2025

Uh oh!

Copilot AI Sep 2, 2025

Uh oh!

Copilot AI Sep 2, 2025

Uh oh!

Uh oh!

Copilot AI Sep 2, 2025

Uh oh!

Uh oh!

	throw;
	// Swallow the exception to avoid throwing from DisposeAsync

		if (patternIndex < pattern.Length && (pattern[patternIndex] == '?' \|\|
		string.Equals(pattern[patternIndex].ToString(), input[inputIndex].ToString(), comparison)))

Implement Virtual FileSystem abstraction over ManagedCode.Storage #107

Implement Virtual FileSystem abstraction over ManagedCode.Storage #107

Uh oh!

Conversation

Copilot AI commented Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Key Features

Usage Example

Architecture

Uh oh!

Copilot AI commented Sep 1, 2025

Uh oh!

KSemenenko commented Sep 1, 2025

Uh oh!

KSemenenko commented Sep 1, 2025

Uh oh!

Copilot AI commented Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

KSemenenko commented Sep 1, 2025

Uh oh!

Copilot AI commented Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Key Changes

Reviewed Changes

Uh oh!

Copilot AI Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI commented Sep 1, 2025 •

edited

Loading

Copilot AI commented Sep 1, 2025 •

edited

Loading

Copilot AI commented Sep 1, 2025 •

edited

Loading