diff --git a/.github/CLAUDE.md b/.github/CLAUDE.md new file mode 100644 index 00000000..da1bb1a1 --- /dev/null +++ b/.github/CLAUDE.md @@ -0,0 +1,230 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +LogExpert is a Windows log file viewer and analyzer built with C# and Windows Forms. It's a GUI replacement for the Unix `tail` command with extensive features including tail mode, filtering, bookmarks, highlighting, and a plugin-based architecture for parsing custom log formats. + +**Key Technologies:** +- .NET 10.0 (Windows target framework) +- Windows Forms for UI +- Nuke Build System with MSBuild +- NUnit for testing +- Plugin-based architecture + +## Build Commands + +### Using Nuke Build (Recommended) + +```powershell +# Build the solution +./build.ps1 + +# Clean and build +./build.ps1 --target Clean Compile + +# Run tests +./build.ps1 --target Test + +# Full release build with packages +./build.ps1 --target Clean Pack CreateSetup --configuration Release +``` + +### Using .NET CLI Directly + +```bash +# From src/ directory +dotnet restore +dotnet build --no-restore +dotnet test --no-build --verbosity normal + +# Run specific test project +dotnet test src/LogExpert.Tests/LogExpert.Tests.csproj +``` + +### Important Build Notes + +- **Windows-only**: This project requires Windows and .NET 10.0.100 SDK (specified in [global.json](global.json)) +- **Cannot build on Linux/macOS**: Uses Windows Desktop SDK and Windows Forms +- Nuke build automatically downloads the correct .NET SDK version if needed +- Output directory: `bin/(Debug|Release)/` + +## Architecture + +### High-Level Structure + +The codebase follows a modular architecture with clear separation of concerns: + +``` +LogExpert/ +├── LogExpert/ # Main application entry point and UI orchestration +├── LogExpert.Core/ # Core business logic, log reading, filtering +├── LogExpert.UI/ # Windows Forms UI components and dialogs +├── LogExpert.Resources/ # Localization resources +├── LogExpert.Configuration/# Configuration management +├── ColumnizerLib/ # Plugin interface definitions +├── PluginRegistry/ # Plugin discovery and security +└── Columnizers/ # Built-in columnizer plugins (CSV, JSON, Regex, etc.) +``` + +### Key Architectural Components + +#### 1. Single Instance Mode with IPC +- Application uses a Mutex to ensure single instance per session +- Named pipes (`LogExpertInstanceMutex{sessionId}`) for inter-process communication +- Secondary instances send file paths to primary instance via JSON over named pipes +- See [Program.cs](src/LogExpert/Program.cs) for implementation + +#### 2. Plugin System +- **Columnizers** (`ILogLineColumnizer`): Parse log lines into columns +- **File System Plugins** (`IFileSystemPlugin`): Support non-local file sources (e.g., SFTP) +- **Context Menu Plugins** (`IContextMenuEntry`): Add custom menu items +- **Keyword Actions** (`IKeywordAction`): React to keywords in logs +- Plugin discovery happens at startup via `PluginRegistry` +- Plugins are loaded from `plugins/` and `pluginsx86/` directories +- Security: Plugin hashes are verified against generated hashes (Release builds only) +- See [PLUGIN_DEVELOPMENT_GUIDE.md](src/docs/PLUGIN_DEVELOPMENT_GUIDE.md) for details + +#### 3. Log File Reading +- Abstract base class: `PositionAwareStreamReaderBase` +- Implementations: `PositionAwareStreamReaderSystem`, `PositionAwareStreamReaderLegacy` +- Uses buffered streams for efficient reading of large files +- Supports encoding detection (UTF-8, UTF-16, UTF-32 with BOM) +- Position tracking for tail mode and seeking +- See [src/LogExpert.Core/Classes/Log/](src/LogExpert.Core/Classes/Log/) for implementations + +#### 4. Configuration Management +- Centralized via `ConfigManager.Instance` +- Initialized with application startup path and screen information +- Supports import/export of settings +- Persists user preferences, columnizer history, highlight masks, etc. +- Configuration stored in application startup directory (portable mode) + +#### 5. Windows Forms UI Architecture +- MDI interface with tab support via `AbstractLogTabWindow` +- Main window created in [LogTabWindow.cs](src/LogExpert.UI/Dialogs/LogTabWindow/) +- Custom controls: `BufferedDataGridView`, `LogTabControl`, `DateTimeDragControl` +- High DPI considerations: Avoid `AutoScaleMode` and `AutoScaleDimensions` on individual controls +- Dark mode support via `Application.SetColorMode()` + +### Critical Files and Their Purposes + +- [Program.cs](src/LogExpert/Program.cs) - Application entry point, IPC setup, single instance handling +- [AbstractLogTabWindow.cs](src/LogExpert.UI/Extensions/LogWindow/AbstractLogTabWindow.cs) - Main window factory and orchestration +- [ILogLineColumnizer.cs](src/ColumnizerLib/ILogLineColumnizer.cs) - Core plugin interface for columnizers +- [ColumnizerPicker.cs](src/LogExpert.Core/Classes/Columnizer/ColumnizerPicker.cs) - Automatic columnizer detection +- [PluginRegistry.cs](src/PluginRegistry/) - Plugin discovery and security verification +- [ConfigManager.cs](src/LogExpert.Configuration/) - Configuration persistence and management +- [LogBuffer.cs](src/LogExpert.Core/Classes/Log/LogBuffer.cs) - In-memory log line caching + +## Development Workflow + +### Adding a New Columnizer Plugin + +1. Create new project in `src/` following naming pattern `*Columnizer` +2. Add project reference to `ColumnizerLib` +3. Implement `ILogLineColumnizer` interface +4. Add project to `src/LogExpert.sln` +5. Create corresponding test project in `Tests/` folder +6. Plugin will be auto-discovered at runtime from output directory + +### Modifying Core Log Reading Logic + +- Core reading classes are in [src/LogExpert.Core/Classes/Log/](src/LogExpert.Core/Classes/Log/) +- Inherit from `PositionAwareStreamReaderBase` for custom stream readers +- Key methods to implement: `ReadLine()`, `Position` property, `Seek()` +- Always maintain position tracking for tail mode support + +### Working with Windows Forms UI + +- UI components in `LogExpert.UI` project +- Follow existing High DPI patterns (no AutoScale on controls) +- Test with both light and dark mode (see `SetDarkMode()` in Program.cs) +- Use localization resources from `LogExpert.Resources` project +- Windows Forms designer files: `*.designer.cs` + +### Testing + +- Unit tests use NUnit framework with Moq for mocking +- Test projects follow naming pattern `*.Tests` +- Test data stored in `TestData/` directories within test projects +- Run all tests: `./build.ps1 --target Test` +- Run specific test: `dotnet test src/LogExpert.Tests/LogExpert.Tests.csproj` + +## Important Patterns and Conventions + +### Code Style +- Nullable reference types enabled (`enable`) +- Comprehensive `.editorconfig` with 4000+ rules +- ImplicitUsings enabled +- Assembly signing enabled (Key.snk) + +### Configuration Files +- **Directory.Build.props** - Common MSBuild properties for all projects +- **Directory.Packages.props** - Centralized NuGet package version management +- **global.json** - .NET SDK version pinning (10.0.100) +- **.editorconfig** - Code style and analysis rules + +### Project Organization +- Solution folders: "Columnizers", "Tests", "docs", "setup" +- Test projects nested under "Tests" solution folder +- Columnizer projects nested under "Columnizers" solution folder +- Documentation in `src/docs/` included in solution + +### Git Workflow +- Default branch: `Development` (use for PRs) +- Commit format: Include "Co-Authored-By: Claude Sonnet 4.5 " +- GitHub Actions run on push to Development branch +- AppVeyor for CI builds and artifact creation + +## Plugin Security System + +**Release builds only:** +- After compilation, `PluginHashGenerator.Tool` generates SHA256 hashes of all plugins +- Hashes stored in [PluginHashGenerator.Generated.cs](src/PluginRegistry/PluginHashGenerator.Generated.cs) +- At runtime, `PluginRegistry` verifies plugin hashes before loading +- Users can trust new plugins via UI dialog +- Hash updates automated via GitHub Actions on successful builds +- See [PLUGIN_HASH_MANAGEMENT.md](src/docs/PLUGIN_HASH_MANAGEMENT.md) + +## Common Gotchas + +1. **Cross-platform builds fail**: This is Windows-only. Don't attempt Linux/macOS builds. +2. **SDK version mismatch**: Must use .NET 10.0.100 SDK (specified in global.json) +3. **Plugin not loading**: Check output directory - plugins must be in `plugins/` or `pluginsx86/` +4. **High DPI issues**: Never use AutoScaleMode on individual controls, only on forms +5. **IPC failures**: Named pipes require proper Windows permissions and session isolation +6. **Encoding detection**: BOM-less files default to encoding from EncodingOptions +7. **Plugin hashes**: Only verified in Release builds; Debug builds skip verification + +## Key Dependencies + +- **NLog**: Logging framework +- **Newtonsoft.Json**: JSON serialization (legacy, but widely used) +- **CsvHelper**: CSV parsing in CsvColumnizer +- **SSH.NET**: SFTP support in SftpFileSystem plugins +- **DockPanelSuite**: Docking panel UI controls +- **Moq/NUnit**: Testing frameworks + +## References + +- Main README: [README.md](README.md) +- Plugin Development: [PLUGIN_DEVELOPMENT_GUIDE.md](src/docs/PLUGIN_DEVELOPMENT_GUIDE.md) +- Plugin Hash System: [PLUGIN_HASH_MANAGEMENT.md](src/docs/PLUGIN_HASH_MANAGEMENT.md) +- Performance Benchmarks: [BENCHMARK_SUMMARY.md](src/docs/performance/BENCHMARK_SUMMARY.md) +- GitHub Wiki: https://github.com/LogExperts/LogExpert/wiki +- Discord: https://discord.gg/SjxkuckRe9 + +# Update Rules File +To update this file, ensure that all sections are kept current with the latest architectural decisions, build processes, and development workflows. Follow these guidelines: +- Review and update build commands if there are changes in the build system. +- Reflect any architectural changes in the "Architecture" section. +- Keep development workflow steps accurate for new contributors. +- Regularly verify links to other documentation files. +- Maintain clarity and conciseness for ease of understanding by new developers. +- Use consistent formatting throughout the document. +- Add new sections as needed for significant changes in the project structure or processes. +- Ensure all technical terms are explained or linked to relevant documentation. +- Periodically review for outdated information and remove or update as necessary. +- If told to not do something, ensure this is also added to the "Dont Do that" section. \ No newline at end of file diff --git a/.gitignore b/.gitignore index 28329487..feaa20e3 100644 --- a/.gitignore +++ b/.gitignore @@ -33,8 +33,8 @@ x86/ bld/ [Bb]in/ [Oo]bj/ -[Ll]og/ -[Ll]ogs/ +/[Ll]og/ +/[Ll]ogs/ # Visual Studio 2015/2017 cache/options directory .vs/ @@ -618,3 +618,102 @@ temp.txt /Data /.tmp /.nuke/temp/*.log + +# roots docs folder should be empty +/docs/* + +# test project logs should be ignored, this is for rolling logfile tests +/src/tools/LogRotator/logs/* + +######## START JET BRAINS git ignore https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore +/src/.idea/* + +# Covers JetBrains IDEs: IntelliJ, GoLand, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider +# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839 + +# User-specific stuff +.idea/**/workspace.xml +.idea/**/tasks.xml +.idea/**/usage.statistics.xml +.idea/**/dictionaries +.idea/**/shelf + +# AWS User-specific +.idea/**/aws.xml + +# Generated files +.idea/**/contentModel.xml + +# Sensitive or high-churn files +.idea/**/dataSources/ +.idea/**/dataSources.ids +.idea/**/dataSources.local.xml +.idea/**/sqlDataSources.xml +.idea/**/dynamic.xml +.idea/**/uiDesigner.xml +.idea/**/dbnavigator.xml + +# Gradle +.idea/**/gradle.xml +.idea/**/libraries + +# Gradle and Maven with auto-import +# When using Gradle or Maven with auto-import, you should exclude module files, +# since they will be recreated, and may cause churn. Uncomment if using +# auto-import. +# .idea/artifacts +# .idea/compiler.xml +# .idea/jarRepositories.xml +# .idea/modules.xml +# .idea/*.iml +# .idea/modules +# *.iml +# *.ipr + +# CMake +cmake-build-*/ + +# Mongo Explorer plugin +.idea/**/mongoSettings.xml + +# File-based project format +*.iws + +# IntelliJ +out/ + +# mpeltonen/sbt-idea plugin +.idea_modules/ + +# JIRA plugin +atlassian-ide-plugin.xml + +# Cursive Clojure plugin +.idea/replstate.xml + +# SonarLint plugin +.idea/sonarlint/ +# see https://community.sonarsource.com/t/is-the-file-idea-idea-idea-sonarlint-xml-intended-to-be-under-source-control/121119 +.idea/sonarlint.xml + +# Crashlytics plugin (for Android Studio and IntelliJ) +com_crashlytics_export_strings.xml +crashlytics.properties +crashlytics-build.properties +fabric.properties + +# Editor-based HTTP Client +.idea/httpRequests +http-client.private.env.json + +# Android studio 3.1+ serialized cache file +.idea/caches/build_file_checksums.ser + +# Apifox Helper cache +.idea/.cache/.Apifox_Helper +.idea/ApifoxUploaderProjectSetting.xml + +# Github Copilot persisted session migrations, see: https://github.com/microsoft/copilot-intellij-feedback/issues/712#issuecomment-3322062215 +.idea/**/copilot.data.migration.*.xml + +######## END JET BRAINS git ignore https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore \ No newline at end of file diff --git a/.vscode/settings.json b/.vscode/settings.json new file mode 100644 index 00000000..5edcd1f3 --- /dev/null +++ b/.vscode/settings.json @@ -0,0 +1,19 @@ +{ + "chat.tools.terminal.autoApprove": { + "dotnet build": true, + "dotnet test": true, + "dotnet clean": true, + "Out-Null": true, + "ForEach-Object": true, + "Set-Content": true, + "Start-Process": true, + "Test-Path": true, + "type": true, + "New-Item": true, + "/^git stash; dotnet test src/LogExpert\\.Tests/LogExpert\\.Tests\\.csproj --filter \"TestShiftBuffers1\" --no-restore 2>&1 \\| Select-Object -Last 30$/": { + "approve": true, + "matchCommandLine": true + }, + "git rev-parse": true + } +} \ No newline at end of file diff --git a/src/AutoColumnizer/AutoColumnizer.cs b/src/AutoColumnizer/AutoColumnizer.cs index 4bb349d9..d9670ca8 100644 --- a/src/AutoColumnizer/AutoColumnizer.cs +++ b/src/AutoColumnizer/AutoColumnizer.cs @@ -62,5 +62,9 @@ public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, st { } + public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, ReadOnlyMemory oldValue) + { + } + #endregion ILogLineColumnizer implementation } \ No newline at end of file diff --git a/src/ColumnizerLib/ILogLineMemoryColumnizer.cs b/src/ColumnizerLib/ILogLineMemoryColumnizer.cs index 05985fb5..fe0db173 100644 --- a/src/ColumnizerLib/ILogLineMemoryColumnizer.cs +++ b/src/ColumnizerLib/ILogLineMemoryColumnizer.cs @@ -94,5 +94,17 @@ public interface ILogLineMemoryColumnizer /// The previous value that was associated with the specified column before the update. void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, string oldValue); + // + /// + /// The callback interface that receives the value update notification. Cannot be null. + /// + /// The zero-based index of the column for which the value is being updated. + /// The new value to be associated with the specified column. + /// + /// The previous value associated with the specified + /// column before the update. + /// + void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, ReadOnlyMemory oldValue); + #endregion } \ No newline at end of file diff --git a/src/CsvColumnizer/CsvColumnizer.cs b/src/CsvColumnizer/CsvColumnizer.cs index 4e3639e8..2f2d4649 100644 --- a/src/CsvColumnizer/CsvColumnizer.cs +++ b/src/CsvColumnizer/CsvColumnizer.cs @@ -164,6 +164,11 @@ public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, st throw new NotImplementedException(); } + public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, ReadOnlyMemory oldValue) + { + throw new NotImplementedException(); + } + public void Selected (ILogLineMemoryColumnizerCallback callback) { ArgumentNullException.ThrowIfNull(callback, nameof(callback)); diff --git a/src/GlassfishColumnizer/GlassfishColumnizer.cs b/src/GlassfishColumnizer/GlassfishColumnizer.cs index 46df4e83..f4c213aa 100644 --- a/src/GlassfishColumnizer/GlassfishColumnizer.cs +++ b/src/GlassfishColumnizer/GlassfishColumnizer.cs @@ -71,12 +71,18 @@ public string[] GetColumnNames () /// /// Creates a new log line instance with text formatted for clipboard copying. /// - /// The returned log line replaces separator characters in the original line with the '|' - /// character to ensure compatibility with clipboard operations. + /// + /// The returned log line replaces separator characters in the original line with the '|' character to ensure + /// compatibility with clipboard operations. + /// /// The log line to be formatted for clipboard use. Cannot be null. - /// A callback interface for columnizer operations. This parameter is reserved for future use and is not utilized in - /// this method. - /// A new instance containing the clipboard-formatted text of the specified log line. + /// + /// A callback interface for columnizer operations. This parameter is reserved for future use and is not utilized in + /// this method. + /// + /// + /// A new instance containing the clipboard-formatted text of the specified log line. + /// public ILogLineMemory GetLineTextForClipboard (ILogLineMemory logLine, ILogLineMemoryColumnizerCallback callback) { return new GlassFishLogLine(ReplaceInMemory(logLine.FullLine, SEPARATOR_CHAR, '|'), logLine.Text, logLine.LineNumber); @@ -85,13 +91,19 @@ public ILogLineMemory GetLineTextForClipboard (ILogLineMemory logLine, ILogLineM /// /// Parses a log line into its constituent columns according to the columnizer's format. /// - /// If the input line does not conform to the expected format or is too short, only the log - /// message column is populated and date/time columns are left blank. The method is tolerant of malformed input and - /// will not throw for common formatting issues. - /// A callback interface used to provide context or services required during columnization. + /// + /// If the input line does not conform to the expected format or is too short, only the log message column is + /// populated and date/time columns are left blank. The method is tolerant of malformed input and will not throw for + /// common formatting issues. + /// + /// + /// A callback interface used to provide context or services required during columnization. + /// /// The log line to be split into columns. - /// An object representing the columnized log line, with each column populated based on the input line. If the line - /// does not match the expected format, the entire line is placed in the log message column. + /// + /// An object representing the columnized log line, with each column populated based on the input line. If the line + /// does not match the expected format, the entire line is placed in the log message column. + /// [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Intentionally passed")] public IColumnizedLogLineMemory SplitLine (ILogLineMemoryColumnizerCallback callback, ILogLineMemory logLine) { @@ -173,8 +185,9 @@ FormatException or /// /// The memory to split /// The separator character - /// Array with 2 elements: [before separator, after separator]. - /// If separator not found, returns [input, Empty] + /// + /// Array with 2 elements: [before separator, after separator]. If separator not found, returns [input, Empty] + /// private static ReadOnlyMemory[] SplitIntoTwo (ReadOnlyMemory input, char separator) { var span = input.Span; @@ -212,15 +225,23 @@ public int GetTimeOffset () /// /// Extracts the timestamp from the specified log line using the expected GlassFish log format. /// - /// The method expects the log line to contain a timestamp in a specific format, typically used - /// by GlassFish logs. If the log line does not match the expected format or the timestamp cannot be parsed, the - /// method returns DateTime.MinValue. - /// A callback interface for columnizer operations. This parameter is not used by this method but is required by the - /// interface. - /// The log line from which to extract the timestamp. Must not be null and should contain a timestamp in the - /// expected format. - /// A DateTime value representing the parsed timestamp from the log line. Returns DateTime.MinValue if the timestamp - /// cannot be extracted or parsed. + /// + /// The method expects the log line to contain a timestamp in a specific format, typically used by GlassFish logs. + /// If the log line does not match the expected format or the timestamp cannot be parsed, the method returns + /// DateTime.MinValue. + /// + /// + /// A callback interface for columnizer operations. This parameter is not used by this method but is required by the + /// interface. + /// + /// + /// The log line from which to extract the timestamp. Must not be null and should contain a timestamp in the + /// expected format. + /// + /// + /// A DateTime value representing the parsed timestamp from the log line. Returns DateTime.MinValue if the timestamp + /// cannot be extracted or parsed. + /// public DateTime GetTimestamp (ILogLineMemoryColumnizerCallback callback, ILogLineMemory logLine) { var temp = logLine.FullLine; @@ -268,23 +289,59 @@ public DateTime GetTimestamp (ILogLineMemoryColumnizerCallback callback, ILogLin /// Updates the internal time offset based on the difference between the specified new and old values when the /// column index is zero. /// - /// If the column index is not zero, this method performs no action. For column 0, both value and - /// oldValue must be valid date and time strings in the required format; otherwise, the time offset is not - /// updated. - /// The callback interface for columnizer operations. This parameter is not used in this method but may be required - /// for interface compatibility. - /// The zero-based index of the column to update. Only a value of 0 triggers a time offset update. - /// The new value to apply. For column 0, this should be a date and time string in the expected format. - /// The previous value to compare against. For column 0, this should be a date and time string in the expected - /// format. + /// + /// If the column index is not zero, this method performs no action. For column 0, both value and oldValue must be + /// valid date and time strings in the required format; otherwise, the time offset is not updated. + /// + /// + /// The callback interface for columnizer operations. This parameter is not used in this method but may be required + /// for interface compatibility. + /// + /// + /// The zero-based index of the column to update. Only a value of 0 triggers a time offset update. + /// + /// + /// The new value to apply. For column 0, this should be a date and time string in the expected format. + /// + /// + /// The previous value to compare against. For column 0, this should be a date and time string in the expected + /// format. + /// public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, string oldValue) + { + PushValue(callback, column, value, oldValue.AsMemory()); + } + + /// + /// Updates the internal time offset based on the difference between the specified new and old values when the + /// column index is zero. + /// + /// + /// If the column index is not zero, this method performs no action. For column 0, both value and oldValue must be + /// valid date and time strings in the required format; otherwise, the time offset is not updated. + /// + /// + /// The callback interface for columnizer operations. This parameter is not used in this method but may be required + /// for interface compatibility. + /// + /// + /// The zero-based index of the column to update. Only a value of 0 triggers a time offset update. + /// + /// + /// The new value to apply. For column 0, this should be a date and time string in the expected format. + /// + /// + /// The previous value to compare against. For column 0, this should be a date and time string in the expected + /// format. + /// + public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, ReadOnlyMemory oldValue) { if (column == 0) { try { var newDateTime = DateTime.ParseExact(value, DATETIME_FORMAT_OUT, _cultureInfo); - var oldDateTime = DateTime.ParseExact(oldValue, DATETIME_FORMAT_OUT, _cultureInfo); + var oldDateTime = DateTime.ParseExact(oldValue.ToString(), DATETIME_FORMAT_OUT, _cultureInfo); var mSecsOld = oldDateTime.Ticks / TimeSpan.TicksPerMillisecond; var mSecsNew = newDateTime.Ticks / TimeSpan.TicksPerMillisecond; _timeOffset = (int)(mSecsNew - mSecsOld); diff --git a/src/JsonColumnizer/JsonColumnizer.cs b/src/JsonColumnizer/JsonColumnizer.cs index 7eda8ccb..26af0ae4 100644 --- a/src/JsonColumnizer/JsonColumnizer.cs +++ b/src/JsonColumnizer/JsonColumnizer.cs @@ -166,6 +166,11 @@ public virtual void PushValue (ILogLineMemoryColumnizerCallback callback, int co throw new NotImplementedException(); } + public virtual void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, ReadOnlyMemory oldValue) + { + throw new NotImplementedException(); + } + public virtual void Selected (ILogLineMemoryColumnizerCallback callback) { ArgumentNullException.ThrowIfNull(callback, nameof(callback)); diff --git a/src/Log4jXmlColumnizer/Log4jXmlColumnizer.cs b/src/Log4jXmlColumnizer/Log4jXmlColumnizer.cs index ecec912b..ffd8f5ce 100644 --- a/src/Log4jXmlColumnizer/Log4jXmlColumnizer.cs +++ b/src/Log4jXmlColumnizer/Log4jXmlColumnizer.cs @@ -213,13 +213,18 @@ public DateTime GetTimestamp (ILogLineMemoryColumnizerCallback callback, ILogLin } public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, string oldValue) + { + PushValue(callback, column, value, oldValue.AsMemory()); + } + + public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, ReadOnlyMemory oldValue) { if (column == 0) { try { var newDateTime = DateTime.ParseExact(value, DATETIME_FORMAT, _cultureInfo); - var oldDateTime = DateTime.ParseExact(oldValue, DATETIME_FORMAT, _cultureInfo); + var oldDateTime = DateTime.ParseExact(oldValue.ToString(), DATETIME_FORMAT, _cultureInfo); var mSecsOld = oldDateTime.Ticks / TimeSpan.TicksPerMillisecond; var mSecsNew = newDateTime.Ticks / TimeSpan.TicksPerMillisecond; _timeOffset = (int)(mSecsNew - mSecsOld); diff --git a/src/LogExpert.Benchmarks/BufferIndexBenchmarks.cs b/src/LogExpert.Benchmarks/BufferIndexBenchmarks.cs new file mode 100644 index 00000000..1b808d3e --- /dev/null +++ b/src/LogExpert.Benchmarks/BufferIndexBenchmarks.cs @@ -0,0 +1,191 @@ +using BenchmarkDotNet.Attributes; + +using ColumnizerLib; + +using LogExpert.Benchmarks.Support; +using LogExpert.Core.Classes.Log.Buffers; + +namespace LogExpert.Benchmarks; + +[MemoryDiagnoser] +[RankColumn] +public class BufferIndexBenchmarks : IDisposable +{ + private BufferIndex _index = null!; + private int _totalLines; + + private bool _disposed; + + [Params(100, 1_000, 10_000)] + public int BufferCount { get; set; } + + private const int LINES_PER_BUFFER = 500; + + [GlobalSetup] + public void Setup () + { + _index = new BufferIndex(BufferCount, LINES_PER_BUFFER); + _totalLines = BufferCount * LINES_PER_BUFFER; + + var fakeFileInfo = new FakeLogFileInfo(); + + using (var writeLock = _index.AcquireWriteLock()) + { + for (int i = 0; i < BufferCount; i++) + { + var buffer = new LogBuffer(fakeFileInfo, LINES_PER_BUFFER) + { + StartLine = i * LINES_PER_BUFFER + }; + + for (int j = 0; j < LINES_PER_BUFFER; j++) + { + buffer.AddLine(new LogLine($"line {i * LINES_PER_BUFFER + j}".AsMemory(), i * LINES_PER_BUFFER + j), 0); + } + + _index.Add(buffer); + } + } + + // Validate setup + var snapshot = _index.CreateSnapshot(); + if (snapshot.BufferCount != BufferCount) + { + throw new InvalidOperationException($"Setup failed: expected {BufferCount} buffers, got {snapshot.BufferCount}"); + } + } + + [GlobalCleanup] + public void Cleanup () => _index.Dispose(); + + /// + /// Simulates tail-follow: reading the last 1000 lines sequentially. + /// Should hit Layer 0 (thread-local cache) ~99% of the time. + /// + [Benchmark(Baseline = true)] + public LogBuffer? SequentialAccess () + { + using var readlock = _index.AcquireReadLock(); + LogBuffer? last = null; + var start = Math.Max(0, _totalLines - 1000); + for (int i = start; i < _totalLines; i++) + { + var logBufferEntry = _index.TryFindBuffer(i); + if (logBufferEntry.Found) + { + last = logBufferEntry.Buffer; + } + } + + return last; + } + + /// + /// Simulates search/goto: deterministic stride across the full file. + /// Co-prime stride visits buffers in non-sequential, non-repeating order. + /// Exercises Layers 2 and 3 heavily. + /// + [Benchmark] + public LogBuffer? StrideAccess () + { + using var readLock = _index.AcquireReadLock(); + LogBuffer? last = null; + var stride = _totalLines / 3 + 1; + var lineNum = 0; + for (int i = 0; i < 1000; i++) + { + var logBufferEntry = _index.TryFindBuffer(lineNum); + if (logBufferEntry.Found) + { + last = logBufferEntry.Buffer; + } + + lineNum = (lineNum + stride) % _totalLines; + } + + return last; + } + + /// + /// Worst case for Layer 0: always crossing buffer boundaries. + /// Exercises Layer 1 (adjacent prediction). + /// + [Benchmark] + public LogBuffer? BoundaryAccess () + { + using var readLock = _index.AcquireReadLock(); + LogBuffer? last = null; + + for (int i = 0; i < 1000; i++) + { + int lineNum = i * (_totalLines / 1000); + var logBufferEntry = _index.TryFindBuffer(lineNum); + if (logBufferEntry.Found) + { + last = logBufferEntry.Buffer; + } + } + + return last; + } + + /// + /// Simulates UI scrolling: page-sized jumps forward through the file. + /// 50-line pages with 3x page jumps (fast scroll drag). + /// Exercises Layer 0 within pages and Layers 1-2 on transitions. + /// + [Benchmark] + public LogBuffer? ScrollAccess () + { + using var readLock = _index.AcquireReadLock(); + LogBuffer? last = null; + const int pageSize = 50; + const int pageJump = pageSize * 3; + var pageStart = 0; + + for (int page = 0; page < 20 && pageStart < _totalLines; page++) + { + var pageEnd = Math.Min(pageStart + pageSize, _totalLines); + for (int line = pageStart; line < pageEnd; line++) + { + var logBufferEntry = _index.TryFindBuffer(line); + if (logBufferEntry.Found) + { + last = logBufferEntry.Buffer; + } + } + + pageStart += pageJump; + } + + return last; + } + + /// + /// Measures LRU eviction cost at current scale. + /// + [Benchmark] + public void EvictAndRepopulate () + { + _index.EvictLeastRecentlyUsed(); + } + + public void Dispose () + { + Dispose(true); + GC.SuppressFinalize(this); + } + + protected virtual void Dispose (bool disposing) + { + if (!_disposed) + { + if (disposing) + { + _index?.Dispose(); + } + + _disposed = true; + } + } +} \ No newline at end of file diff --git a/src/LogExpert.Benchmarks/BufferIndexContentionBenchmarks.cs b/src/LogExpert.Benchmarks/BufferIndexContentionBenchmarks.cs new file mode 100644 index 00000000..9885ddeb --- /dev/null +++ b/src/LogExpert.Benchmarks/BufferIndexContentionBenchmarks.cs @@ -0,0 +1,170 @@ +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Diagnosers; + +using ColumnizerLib; + +using LogExpert.Benchmarks.Support; +using LogExpert.Core.Classes.Log.Buffers; + +namespace LogExpert.Benchmarks; + +/// +/// Measures ReaderWriterLockSlim contention under concurrent read load. +/// Compares single-threaded throughput against N concurrent readers +/// to determine if RWLS is a bottleneck worth optimizing. +/// +[MemoryDiagnoser] +[ThreadingDiagnoser] // Reports lock contention + thread pool stats +[RankColumn] +public class BufferIndexContentionBenchmarks : IDisposable +{ + private BufferIndex _index = null!; + private int _totalLines; + private bool _disposed; + + private const int BUFFERS = 10_000; + private const int LINES_PER_BUFFER = 500; + private const int READS_PER_TASK = 1_000; + + [GlobalSetup] + public void Setup () + { + _index = new BufferIndex(BUFFERS, LINES_PER_BUFFER); + _totalLines = BUFFERS * LINES_PER_BUFFER; + + var fakeFileInfo = new FakeLogFileInfo(); + using var writeLock = _index.AcquireWriteLock(); + for (int i = 0; i < BUFFERS; i++) + { + var buffer = new LogBuffer(fakeFileInfo, LINES_PER_BUFFER) + { + StartLine = i * LINES_PER_BUFFER + }; + for (int j = 0; j < LINES_PER_BUFFER; j++) + { + buffer.AddLine( + new LogLine($"line {i * LINES_PER_BUFFER + j}".AsMemory(), + i * LINES_PER_BUFFER + j), 0); + } + _index.Add(buffer); + } + } + + /// + /// Single-threaded baseline: sequential reads under one read lock. + /// This is the ideal throughput ceiling. + /// + [Benchmark(Baseline = true)] + public int SingleThreadedReads () + { + int found = 0; + using var readLock = _index.AcquireReadLock(); + var start = Math.Max(0, _totalLines - READS_PER_TASK); + for (int i = start; i < _totalLines; i++) + { + if (_index.TryFindBuffer(i).Found) + { + found++; + } + } + + return found; + } + + /// + /// N concurrent readers each acquiring their own read lock. + /// If RWLS has no contention, throughput ≈ N × single-threaded. + /// + [Benchmark] + [Arguments(2)] + [Arguments(4)] + [Arguments(8)] + [Arguments(12)] + public int ConcurrentReads (int threadCount) + { + var total = 0; + _ = Parallel.For(0, threadCount, _ => + { + int found = 0; + using var readLock = _index.AcquireReadLock(); + var start = Math.Max(0, _totalLines - READS_PER_TASK); + for (int i = start; i < _totalLines; i++) + { + if (_index.TryFindBuffer(i).Found) + { + found++; + } + } + _ = Interlocked.Add(ref total, found); + }); + return total; + } + + /// + /// Simulates production: N readers + 1 writer (tail-follow append). + /// Writer acquires write lock briefly every ~1000 reads. + /// This is the realistic contention scenario. + /// + [Benchmark] + [Arguments(4)] + [Arguments(8)] + public int ConcurrentReadsWithWriter (int readerCount) + { + using var cts = new CancellationTokenSource(); + var total = 0; + + // Writer task: periodically takes write lock (simulates new buffer append) + var writerTask = Task.Run(() => + { + while (!cts.Token.IsCancellationRequested) + { + using var writeLock = _index.AcquireWriteLock(); + // Simulate brief write work (no actual mutation to keep state clean) + Thread.SpinWait(100); + } + }); + + // Reader tasks + _ = Parallel.For(0, readerCount, _ => + { + int found = 0; + using var readLock = _index.AcquireReadLock(); + var start = Math.Max(0, _totalLines - READS_PER_TASK); + for (int i = start; i < _totalLines; i++) + { + if (_index.TryFindBuffer(i).Found) + { + found++; + } + } + + _ = Interlocked.Add(ref total, found); + }); + + cts.Cancel(); + writerTask.Wait(); + return total; + } + + [GlobalCleanup] + public void Cleanup () => _index.Dispose(); + + public void Dispose () + { + Dispose(true); + GC.SuppressFinalize(this); + } + + protected virtual void Dispose (bool disposing) + { + if (!_disposed) + { + if (disposing) + { + _index?.Dispose(); + } + + _disposed = true; + } + } +} \ No newline at end of file diff --git a/src/LogExpert.Benchmarks/LogExpert.Benchmarks.csproj b/src/LogExpert.Benchmarks/LogExpert.Benchmarks.csproj index 1240cbd4..4bfb4225 100644 --- a/src/LogExpert.Benchmarks/LogExpert.Benchmarks.csproj +++ b/src/LogExpert.Benchmarks/LogExpert.Benchmarks.csproj @@ -16,6 +16,7 @@ + diff --git a/src/LogExpert.Benchmarks/Program.cs b/src/LogExpert.Benchmarks/Program.cs new file mode 100644 index 00000000..01954f88 --- /dev/null +++ b/src/LogExpert.Benchmarks/Program.cs @@ -0,0 +1,54 @@ +using BenchmarkDotNet.Running; + +namespace LogExpert.Benchmarks; + +public static class Program +{ + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Benchmarks")] + public static void Main (string[] args) + { + if (args == null || args.Length == 0) + { + Console.WriteLine("No benchmarks specified. Running all benchmarks..."); + + // Run all benchmarks if no arguments are provided + _ = BenchmarkRunner.Run(); + _ = BenchmarkRunner.Run(); + _ = BenchmarkRunner.Run(); + _ = BenchmarkRunner.Run(); + } + else + { + // Run specific benchmarks based on command-line arguments + _ = BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args); + } + + Console.WriteLine("Replace with the name of the benchmark you want to run, e.g. "); + Console.WriteLine("StreamReaderBenchmarks: Benchmarks for stream readers"); + Console.WriteLine("ReadThroughputBenchmarks: Benchmarks for read throughput"); + Console.WriteLine("BufferIndexBenchmarks: Benchmarks for buffer index"); + Console.WriteLine("BufferIndexContentionBenchmarks: Benchmarks for buffer index contention"); + Console.WriteLine("Dry run:"); + Console.WriteLine("dotnet run -c Release -- --filter \"**\" --job Dry --noOverwrite"); + Console.WriteLine("Short run:"); + Console.WriteLine("dotnet run -c Release -- --filter \"**\" --job Short --noOverwrite"); + Console.WriteLine("Full baseline run:"); + Console.WriteLine("dotnet run -c Release -- --filter \"**\" --noOverwrite"); + } +} + +/* + * Comment / Uncommen the benchmark to run, careful some can run longer + * 1.) a dry run + * dotnet run -c Release -- --filter "StreamReaderBenchmarks" --job Dry --noOverwrite + * 2.) a short run + * dotnet run -c Release -- --filter "StreamReaderBenchmarks" --job Short --noOverwrite + * 3.) a full baseline run + * dotnet run -c Release -- --filter "StreamReaderBenchmarks" --noOverwrite + * + * The full baseline run generates a MD file + * BenchmarkDotNet.Artifacts/results/*-report-github.md + * + * If changes are made with the LogfileReader / BufferIndex, always do a Benchmark to + * verify no performance regression is introduced, especially with large files. + */ diff --git a/src/LogExpert.Benchmarks/ReadThroughputBenchmarks.cs b/src/LogExpert.Benchmarks/ReadThroughputBenchmarks.cs new file mode 100644 index 00000000..688e2bd5 --- /dev/null +++ b/src/LogExpert.Benchmarks/ReadThroughputBenchmarks.cs @@ -0,0 +1,151 @@ +using System.Text; + +using BenchmarkDotNet.Attributes; + +using LogExpert.Core.Classes.Log; +using LogExpert.Core.Entities; +using LogExpert.Core.Enums; + +namespace LogExpert.Benchmarks; + +/// +/// Measures LogfileReader.ReadFiles() throughput with different progress reporters. +/// Uses real temp files to include actual I/O in the measurement. +/// +[MemoryDiagnoser] +[RankColumn] +public class ReadThroughputBenchmarks +{ + private string _tempFile = null!; + + [Params(10_000, 100_000, 1_000_000)] + public int LineCount { get; set; } + + [GlobalSetup] + public void Setup () + { + _tempFile = Path.GetTempFileName(); + GenerateLogFile(_tempFile, LineCount); + + // Initialize PluginRegistry for local file system support + // (or use NullPluginRegistry if constructor doesn't need it) + _ = PluginRegistry.PluginRegistry.Create(Path.GetDirectoryName(_tempFile)!, 500); + } + + /// + /// Baseline: read with NullProgressReporter (zero event overhead). + /// + [Benchmark(Baseline = true)] + public int ReadWithNullReporter () + { + using var reader = new LogfileReader( + _tempFile, + new EncodingOptions { Encoding = Encoding.UTF8 }, + multiFile: false, + bufferCount: 500, + linesPerBuffer: 500, + new MultiFileOptions(), + ReaderType.System, + PluginRegistry.PluginRegistry.Instance, + maximumLineLength: 500, + progressReporter: Core.Classes.Log.ProgressReporters.NullProgressReporter.Instance); + + reader.ReadFiles(); + return reader.LineCount; + } + + /// + /// Production path: read with PeriodicProgressReporter (default, no subscribers). + /// + [Benchmark] + public int ReadWithPeriodicReporter () + { + using var reader = new LogfileReader( + _tempFile, + new EncodingOptions { Encoding = Encoding.UTF8 }, + multiFile: false, + bufferCount: 500, + linesPerBuffer: 500, + new MultiFileOptions(), + ReaderType.System, + PluginRegistry.PluginRegistry.Instance, + maximumLineLength: 500); + // No progressReporter = default PeriodicProgressReporter + + reader.ReadFiles(); + return reader.LineCount; + } + + /// + /// Post-change: read with block-based allocation (System reader uses CharBlockAllocator). + /// Compare Gen0/Gen1/Gen2 collections vs baseline to validate allocation reduction. + /// This method is identical to ReadWithNullReporter — it exists solely for explicit + /// before/after naming in benchmark reports. + /// + [Benchmark] + public int ReadWithBlockAllocation () + { + using var reader = new LogfileReader( + _tempFile, + new EncodingOptions { Encoding = Encoding.UTF8 }, + multiFile: false, + bufferCount: 500, + linesPerBuffer: 500, + new MultiFileOptions(), + ReaderType.System, + PluginRegistry.PluginRegistry.Instance, + maximumLineLength: 500, + progressReporter: Core.Classes.Log.ProgressReporters.NullProgressReporter.Instance); + + reader.ReadFiles(); + return reader.LineCount; + } + + /// + /// Direct-read: reads decoded chars directly into pooled blocks without per-line string allocation. + /// Compare allocations and GC counts vs ReadWithBlockAllocation. + /// + [Benchmark] + public int ReadWithDirectRead () + { + using var reader = new LogfileReader( + _tempFile, + new EncodingOptions { Encoding = Encoding.UTF8 }, + multiFile: false, + bufferCount: 500, + linesPerBuffer: 500, + new MultiFileOptions(), + ReaderType.SystemDirect, + PluginRegistry.PluginRegistry.Instance, + maximumLineLength: 500, + progressReporter: Core.Classes.Log.ProgressReporters.NullProgressReporter.Instance); + + reader.ReadFiles(); + return reader.LineCount; + } + + [GlobalCleanup] + public void Cleanup () + { + if (File.Exists(_tempFile)) + { + File.Delete(_tempFile); + } + } + + [System.Diagnostics.CodeAnalysis.SuppressMessage("Security", "CA5394:Do not use insecure randomness", Justification = "Benchmark data generation")] + private static void GenerateLogFile (string path, int lineCount) + { + var rng = new Random(42); // deterministic seed for reproducibility + using var writer = new StreamWriter(path, false, Encoding.UTF8, bufferSize: 65536); + for (int i = 0; i < lineCount; i++) + { + writer.Write("2026-04-23 12:00:00."); + writer.Write(i % 1000); + writer.Write(" [INFO] Thread-"); + writer.Write(rng.Next(1, 32)); + writer.Write(" SomeNamespace.SomeClass - Log message number "); + writer.WriteLine(i); + } + } +} \ No newline at end of file diff --git a/src/LogExpert.Benchmarks/StreamReaderBenchmarks.cs b/src/LogExpert.Benchmarks/StreamReaderBenchmarks.cs index c8a0b382..a82f0432 100644 --- a/src/LogExpert.Benchmarks/StreamReaderBenchmarks.cs +++ b/src/LogExpert.Benchmarks/StreamReaderBenchmarks.cs @@ -1,9 +1,8 @@ using System.Text; using BenchmarkDotNet.Attributes; -using BenchmarkDotNet.Running; -using LogExpert.Core.Classes.Log; +using LogExpert.Core.Classes.Log.Streamreaders; using LogExpert.Core.Entities; using LogExpert.Core.Interfaces; @@ -150,12 +149,4 @@ private static void ReadAllLines (ILogStreamReader reader) // Consume the line } } -} - -public static class Program -{ - public static void Main (string[] args) - { - _ = BenchmarkRunner.Run(); - } -} +} \ No newline at end of file diff --git a/src/LogExpert.Benchmarks/Support/FakeLogFileInfo.cs b/src/LogExpert.Benchmarks/Support/FakeLogFileInfo.cs new file mode 100644 index 00000000..bd70a382 --- /dev/null +++ b/src/LogExpert.Benchmarks/Support/FakeLogFileInfo.cs @@ -0,0 +1,34 @@ +using ColumnizerLib; + +namespace LogExpert.Benchmarks.Support; + +/// +/// Minimal ILogFileInfo stub for benchmarks. No filesystem access. +/// Wraps an in-memory byte array as the file content. +/// +internal sealed class FakeLogFileInfo : ILogFileInfo +{ + private readonly byte[] _content; + + public FakeLogFileInfo (string name = "fake.log", byte[]? content = null, long length = 1_000_000) + { + FullName = name; + _content = content ?? []; + Length = content?.Length ?? length; + OriginalLength = Length; + } + + public string FullName { get; } + public string FileName => Path.GetFileName(FullName); + public string DirectoryName => Path.GetDirectoryName(FullName) ?? ""; + public char DirectorySeparatorChar => Path.DirectorySeparatorChar; + public Uri Uri => new($"file:///{FullName}"); + public long Length { get; set; } + public long OriginalLength { get; } + public bool FileExists => true; + public int PollInterval => 250; + + public bool FileHasChanged () => false; + public Stream OpenStream () => new MemoryStream(_content, writable: false); + public ILogFileInfo GetRolloverInfo (string fileName) => new FakeLogFileInfo(fileName); +} \ No newline at end of file diff --git a/src/LogExpert.Benchmarks/Support/NullPluginRegistry.cs b/src/LogExpert.Benchmarks/Support/NullPluginRegistry.cs new file mode 100644 index 00000000..5c815054 --- /dev/null +++ b/src/LogExpert.Benchmarks/Support/NullPluginRegistry.cs @@ -0,0 +1,30 @@ +using ColumnizerLib; + +using LogExpert.Core.Interfaces; + +namespace LogExpert.Benchmarks.Support; + +/// +/// No-op IPluginRegistry for benchmarks. Returns empty columnizer list and +/// a stub file system plugin that handles all URIs via local file system. +/// +internal sealed class NullPluginRegistry : IPluginRegistry +{ + public static readonly NullPluginRegistry Instance = new(); + + public IList RegisteredColumnizers { get; } = []; + + public IFileSystemPlugin FindFileSystemForUri (string fileNameOrUri) => NullFileSystemPlugin.Instance; + + private sealed class NullFileSystemPlugin : IFileSystemPlugin + { + public static readonly NullFileSystemPlugin Instance = new(); + + public string Text => "Null"; + public string Description => "No-op file system for benchmarks"; + public bool CanHandleUri (string uriString) => true; + + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "For UnitTests")] + public ILogFileInfo GetLogfileInfo (string uriString) => throw new NotSupportedException("NullFileSystemPlugin does not support GetLogfileInfo"); + } +} \ No newline at end of file diff --git a/src/LogExpert.Core/Classes/Columnizer/ClfColumnizer.cs b/src/LogExpert.Core/Classes/Columnizer/ClfColumnizer.cs index 3d299940..6cfcdda4 100644 --- a/src/LogExpert.Core/Classes/Columnizer/ClfColumnizer.cs +++ b/src/LogExpert.Core/Classes/Columnizer/ClfColumnizer.cs @@ -67,13 +67,19 @@ public string[] GetColumnNames () /// /// Extracts the timestamp from the specified log line using the provided callback. /// - /// If the log line does not contain a valid timestamp in the expected column or format, the - /// method returns DateTime.MinValue. The expected timestamp format and column position are determined by the - /// implementation and may vary depending on the log source. - /// A callback interface used to assist in parsing the log line and retrieving column information. + /// + /// If the log line does not contain a valid timestamp in the expected column or format, the method returns + /// DateTime.MinValue. The expected timestamp format and column position are determined by the implementation and + /// may vary depending on the log source. + /// + /// + /// A callback interface used to assist in parsing the log line and retrieving column information. + /// /// The log line from which to extract the timestamp. - /// A DateTime value representing the timestamp extracted from the log line. Returns DateTime.MinValue if the - /// timestamp cannot be parsed or is not present. + /// + /// A DateTime value representing the timestamp extracted from the log line. Returns DateTime.MinValue if the + /// timestamp cannot be parsed or is not present. + /// public DateTime GetTimestamp (ILogLineMemoryColumnizerCallback callback, ILogLineMemory logLine) { // Use SplitLine to parse, then extract timestamp column @@ -104,14 +110,20 @@ FormatException or /// /// Splits a log line into its constituent columns using the configured columnizer logic. /// - /// If the input line does not match the expected format, the entire line is placed in the - /// request column. For lines longer than 1024 characters, only the first 1024 characters are used for - /// columnization. The method does not localize column values. - /// A callback interface used to provide additional context or services required during columnization. Cannot be - /// null. + /// + /// If the input line does not match the expected format, the entire line is placed in the request column. For lines + /// longer than 1024 characters, only the first 1024 characters are used for columnization. The method does not + /// localize column values. + /// + /// + /// A callback interface used to provide additional context or services required during columnization. Cannot be + /// null. + /// /// The log line to be split into columns. Cannot be null. - /// An object representing the columnized log line, with each column populated according to the parsed content of - /// the input line. + /// + /// An object representing the columnized log line, with each column populated according to the parsed content of + /// the input line. + /// [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Intentionally Passed")] public IColumnizedLogLineMemory SplitLine (ILogLineMemoryColumnizerCallback callback, ILogLineMemory logLine) { @@ -237,20 +249,27 @@ public string GetCustomName () /// /// Processes a value change for a specified column and notifies the callback of the update. /// - /// If the column index is 2, the method attempts to interpret the values as date and time - /// strings and calculates the time offset in milliseconds. No action is taken for other column indices. + /// + /// If the column index is 2, the method attempts to interpret the values as date and time strings and calculates + /// the time offset in milliseconds. No action is taken for other column indices. + /// /// The callback interface used to handle column value updates. /// The zero-based index of the column for which the value is being updated. /// The new value to be set for the specified column. /// The previous value of the specified column before the update. public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, string oldValue) + { + PushValue(callback, column, value, oldValue.AsMemory()); + } + + public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, ReadOnlyMemory oldValue) { if (column == 2) { try { var newDateTime = DateTime.ParseExact(value, DATE_TIME_FORMAT, _cultureInfo); - var oldDateTime = DateTime.ParseExact(oldValue, DATE_TIME_FORMAT, _cultureInfo); + var oldDateTime = DateTime.ParseExact(oldValue.ToString(), DATE_TIME_FORMAT, _cultureInfo); var mSecsOld = oldDateTime.Ticks / TimeSpan.TicksPerMillisecond; var mSecsNew = newDateTime.Ticks / TimeSpan.TicksPerMillisecond; _timeOffset = (int)(mSecsNew - mSecsOld); @@ -264,10 +283,11 @@ public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, st /// /// Provides a compiled regular expression used to parse lines matching a specific log entry format. /// - /// The regular expression is precompiled for performance and is intended to extract fields from - /// log lines with a fixed format. The pattern captures multiple groups, including text fields and quoted values. - /// Use the returned to match and extract data from log entries conforming to this - /// structure. + /// + /// The regular expression is precompiled for performance and is intended to extract fields from log lines with a + /// fixed format. The pattern captures multiple groups, including text fields and quoted values. Use the returned + /// to match and extract data from log entries conforming to this structure. + /// /// A instance that matches lines with the expected log entry structure. [GeneratedRegex("(.*) (-) (.*) (\\[.*\\]) (\".*\") (.*) (.*) (\".*\") (\".*\")")] private static partial Regex LineRegex (); diff --git a/src/LogExpert.Core/Classes/Columnizer/SquareBracketColumnizer.cs b/src/LogExpert.Core/Classes/Columnizer/SquareBracketColumnizer.cs index dce092c9..5ac9f6da 100644 --- a/src/LogExpert.Core/Classes/Columnizer/SquareBracketColumnizer.cs +++ b/src/LogExpert.Core/Classes/Columnizer/SquareBracketColumnizer.cs @@ -6,11 +6,13 @@ namespace LogExpert.Core.Classes.Columnizer; /// Provides functionality to split log lines into columns based on square bracket delimiters, typically extracting /// date, time, and message fields for log analysis. /// -/// This columnizer is designed for log formats where fields are enclosed in square brackets or separated -/// by whitespace, with optional date and time columns at the beginning of each line. It supports dynamic detection of -/// column structure based on sample log lines and can apply a time offset to parsed timestamps. The class implements -/// interfaces for memory-efficient log line processing and columnizer prioritization, making it suitable for -/// integration with log viewers or analysis tools that require flexible column extraction. +/// +/// This columnizer is designed for log formats where fields are enclosed in square brackets or separated by whitespace, +/// with optional date and time columns at the beginning of each line. It supports dynamic detection of column structure +/// based on sample log lines and can apply a time offset to parsed timestamps. The class implements interfaces for +/// memory-efficient log line processing and columnizer prioritization, making it suitable for integration with log +/// viewers or analysis tools that require flexible column extraction. +/// public class SquareBracketColumnizer : ILogLineMemoryColumnizer, IColumnizerPriorityMemory { #region ILogLineMemoryColumnizer implementation @@ -51,7 +53,9 @@ public bool IsTimeshiftImplemented () /// /// Sets the time offset, in milliseconds, to be applied to time calculations. /// - /// The time offset, in milliseconds, to apply. Positive values advance the time; negative values delay it. + /// + /// The time offset, in milliseconds, to apply. Positive values advance the time; negative values delay it. + /// public void SetTimeOffset (int msecOffset) { _timeOffset = msecOffset; @@ -60,8 +64,10 @@ public void SetTimeOffset (int msecOffset) /// /// Gets the current time offset, in seconds, applied to time calculations. /// - /// The time offset, in seconds. A positive value indicates a forward offset; a negative value indicates a backward - /// offset. + /// + /// The time offset, in seconds. A positive value indicates a forward offset; a negative value indicates a backward + /// offset. + /// public int GetTimeOffset () { return _timeOffset; @@ -70,13 +76,16 @@ public int GetTimeOffset () /// /// Extracts and parses the timestamp from the specified log line. /// - /// If the log line does not contain a valid timestamp or if parsing fails, the method returns - /// DateTime.MinValue. The expected timestamp is typically composed of the first two columns in the log - /// line. + /// + /// If the log line does not contain a valid timestamp or if parsing fails, the method returns DateTime.MinValue. + /// The expected timestamp is typically composed of the first two columns in the log line. + /// /// A callback interface used to assist with columnizing the log line. /// The log line from which to extract the timestamp. - /// A DateTime value representing the parsed timestamp if extraction and parsing succeed; otherwise, - /// DateTime.MinValue. + /// + /// A DateTime value representing the parsed timestamp if extraction and parsing succeed; otherwise, + /// DateTime.MinValue. + /// public DateTime GetTimestamp (ILogLineMemoryColumnizerCallback callback, ILogLineMemory logLine) { var cols = SplitLine(callback, logLine); @@ -119,7 +128,9 @@ public string GetName () /// /// Gets a description of the log line splitting format, including the expected fields. /// - /// A string describing how each log line is split into fields: Date, Time, and the remainder of the log message. + /// + /// A string describing how each log line is split into fields: Date, Time, and the remainder of the log message. + /// public string GetDescription () { return "Splits every line into n fields: Date, Time and the rest of the log message"; @@ -137,13 +148,17 @@ public int GetColumnCount () /// /// Returns an array of column names based on the current log format configuration. /// - /// The set and order of column names depend on the log format and configuration. If time - /// information is present, the array includes "Date" and "Time" columns. Additional columns such as "Level" and - /// "Source" are included if the log contains more than three or four columns, respectively. Any extra columns are - /// named sequentially as "Source1", "Source2", etc., before the final "Message" column. - /// An array of strings containing the names of all columns in the log. The array includes standard columns such as + /// + /// The set and order of column names depend on the log format and configuration. If time information is present, + /// the array includes "Date" and "Time" columns. Additional columns such as "Level" and "Source" are included if + /// the log contains more than three or four columns, respectively. Any extra columns are named sequentially as + /// "Source1", "Source2", etc., before the final "Message" column. + /// + /// + /// An array of strings containing the names of all columns in the log. The array includes standard columns such as /// "Date", "Time", "Level", "Source", and "Message", as well as additional source columns if present. The array - /// will contain one element for each column in the log, in the order they appear. + /// will contain one element for each column in the log, in the order they appear. + /// public string[] GetColumnNames () { var columnNames = new List(GetColumnCount()); @@ -178,14 +193,19 @@ public string[] GetColumnNames () /// /// Splits the specified log line into its constituent columns based on detected date and time formats. /// - /// If the log line does not match a recognized date and time format, the entire line is treated - /// as a single column. If the log line is too short to contain date or time information, it is returned as a single - /// column as well. - /// A callback interface that can be used during the columnization process. This parameter may be used to provide - /// additional context or services required for columnization. + /// + /// If the log line does not match a recognized date and time format, the entire line is treated as a single column. + /// If the log line is too short to contain date or time information, it is returned as a single column as well. + /// + /// + /// A callback interface that can be used during the columnization process. This parameter may be used to provide + /// additional context or services required for columnization. + /// /// The log line to be split into columns. Cannot be null. - /// An object representing the columnized version of the input log line. The returned object contains the extracted - /// columns, which may include date, time, and the remainder of the line, depending on the detected format. + /// + /// An object representing the columnized version of the input log line. The returned object contains the extracted + /// columns, which may include date, time, and the remainder of the line, depending on the detected format. + /// [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Intentionally passed")] public IColumnizedLogLineMemory SplitLine (ILogLineMemoryColumnizerCallback callback, ILogLineMemory logLine) { @@ -258,16 +278,22 @@ FormatException or /// /// Splits a log line into an array of columns based on date, time, and bracketed field positions. /// - /// If the input line does not contain enough fields to match the expected column count, empty - /// columns are inserted to ensure the returned array has the correct length. The method associates each column with - /// the provided parent log line object. + /// + /// If the input line does not contain enough fields to match the expected column count, empty columns are inserted + /// to ensure the returned array has the correct length. The method associates each column with the provided parent + /// log line object. + /// /// The log line to split, provided as a read-only memory buffer of characters. /// The length, in characters, of the date field at the start of the line. /// The length, in characters, of the time field following the date field. - /// The zero-based position in the line immediately after the date and time fields. + /// + /// The zero-based position in the line immediately after the date and time fields. + /// /// The parent log line object to associate with each resulting column. - /// An array of columns parsed from the input line. The array contains one element for each expected column, with - /// empty columns inserted if the input does not provide enough fields. + /// + /// An array of columns parsed from the input line. The array contains one element for each expected column, with + /// empty columns inserted if the input does not provide enough fields. + /// private Column[] SquareSplit (ReadOnlyMemory line, int dateLen, int timeLen, int dateTimeEndPos, ColumnizedLogLine clogLine) { List columnList = []; @@ -331,12 +357,16 @@ private Column[] SquareSplit (ReadOnlyMemory line, int dateLen, int timeLe /// Determines the priority level for parsing log lines based on the specified file name and a collection of log /// line samples. /// - /// The returned priority reflects how well the log format is supported based on the structure - /// and content of the provided samples. This method does not modify the input collection. + /// + /// The returned priority reflects how well the log format is supported based on the structure and content of the + /// provided samples. This method does not modify the input collection. + /// /// The name of the log file to analyze. Cannot be null. /// A collection of log line samples to evaluate for format support. Cannot be null. - /// A value indicating the priority level for parsing the provided log lines. Returns a higher priority if the - /// format is well supported or perfectly supported; otherwise, returns a lower priority. + /// + /// A value indicating the priority level for parsing the provided log lines. Returns a higher priority if the + /// format is well supported or perfectly supported; otherwise, returns a lower priority. + /// public Priority GetPriority (string fileName, IEnumerable samples) { ArgumentNullException.ThrowIfNull(fileName, nameof(fileName)); @@ -430,8 +460,12 @@ public Priority GetPriority (string fileName, IEnumerable sample /// /// Determines the priority for processing the specified log file based on the provided log line samples. /// - /// The name of the log file for which to determine the processing priority. Cannot be null or empty. - /// A collection of log line samples used to assess the file's priority. Cannot be null. + /// + /// The name of the log file for which to determine the processing priority. Cannot be null or empty. + /// + /// + /// A collection of log line samples used to assess the file's priority. Cannot be null. + /// /// A value indicating the determined priority for the specified log file. public Priority GetPriority (string fileName, IEnumerable samples) { @@ -442,28 +476,39 @@ public Priority GetPriority (string fileName, IEnumerable samples) /// Processes a value change for a specified column and updates the time offset if the column represents a /// timestamp. /// - /// This method only updates the time offset when the specified column index is 1 and both the - /// new and old values can be parsed as valid timestamps according to the determined time format. No action is taken - /// for other columns or if the values cannot be parsed as dates. - /// The callback interface used to interact with the columnizer during value processing. - /// The zero-based index of the column for which the value is being processed. If the value is 1, the method - /// attempts to update the time offset. + /// + /// This method only updates the time offset when the specified column index is 1 and both the new and old values + /// can be parsed as valid timestamps according to the determined time format. No action is taken for other columns + /// or if the values cannot be parsed as dates. + /// + /// + /// The callback interface used to interact with the columnizer during value processing. + /// + /// + /// The zero-based index of the column for which the value is being processed. If the value is 1, the method + /// attempts to update the time offset. + /// /// The new value to be processed for the specified column. /// The previous value of the specified column before the change. public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, string oldValue) + { + PushValue(callback, column, value, oldValue.AsMemory()); + } + + public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, ReadOnlyMemory oldValue) { if (column == 1) { try { - var formatInfo = _timeFormatDeterminer.DetermineTimeFormatInfo(oldValue.AsSpan()); + var formatInfo = _timeFormatDeterminer.DetermineTimeFormatInfo(oldValue.Span); if (formatInfo == null) { return; } var newDateTime = DateTime.ParseExact(value, formatInfo.TimeFormat, formatInfo.CultureInfo); - var oldDateTime = DateTime.ParseExact(oldValue, formatInfo.TimeFormat, formatInfo.CultureInfo); + var oldDateTime = DateTime.ParseExact(oldValue.Span, formatInfo.TimeFormat, formatInfo.CultureInfo); var mSecsOld = oldDateTime.Ticks / TimeSpan.TicksPerMillisecond; var mSecsNew = newDateTime.Ticks / TimeSpan.TicksPerMillisecond; _timeOffset = (int)(mSecsNew - mSecsOld); diff --git a/src/LogExpert.Core/Classes/Columnizer/TimestampColumnizer.cs b/src/LogExpert.Core/Classes/Columnizer/TimestampColumnizer.cs index f1d2a2d7..3a952aa3 100644 --- a/src/LogExpert.Core/Classes/Columnizer/TimestampColumnizer.cs +++ b/src/LogExpert.Core/Classes/Columnizer/TimestampColumnizer.cs @@ -49,13 +49,18 @@ public string[] GetColumnNames () /// /// Splits a log line into its constituent columns, typically separating date, time, and the remainder of the line. /// - /// If the log line does not match a recognized date/time format, the entire line is returned as - /// a single column. Columns typically represent the date, time, and the rest of the log entry. If parsing fails due - /// to format issues, column values are set to "n/a" except for the remainder, which contains the original - /// line. - /// A callback interface used to provide additional context or services required during columnization. + /// + /// If the log line does not match a recognized date/time format, the entire line is returned as a single column. + /// Columns typically represent the date, time, and the rest of the log entry. If parsing fails due to format + /// issues, column values are set to "n/a" except for the remainder, which contains the original line. + /// + /// + /// A callback interface used to provide additional context or services required during columnization. + /// /// The log line to be split into columns. Cannot be null. - /// An object representing the columnized log line, with each column containing a segment of the original log line. + /// + /// An object representing the columnized log line, with each column containing a segment of the original log line. + /// [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Intentionally passed")] public IColumnizedLogLineMemory SplitLine (ILogLineMemoryColumnizerCallback callback, ILogLineMemory logLine) { @@ -144,13 +149,16 @@ FormatException or /// /// Extracts and parses the timestamp from the specified log line using the provided callback. /// - /// If the log line does not contain a valid timestamp in the expected columns or if parsing - /// fails, the method returns DateTime.MinValue. The timestamp is expected to be composed from the first two columns - /// of the log line. + /// + /// If the log line does not contain a valid timestamp in the expected columns or if parsing fails, the method + /// returns DateTime.MinValue. The timestamp is expected to be composed from the first two columns of the log line. + /// /// The callback used to access column information for the log line. /// The log line from which to extract the timestamp. - /// A DateTime value representing the parsed timestamp if extraction and parsing succeed; otherwise, - /// DateTime.MinValue. + /// + /// A DateTime value representing the parsed timestamp if extraction and parsing succeed; otherwise, + /// DateTime.MinValue. + /// public DateTime GetTimestamp (ILogLineMemoryColumnizerCallback callback, ILogLineMemory logLine) { var cols = SplitLine(callback, logLine); @@ -192,19 +200,24 @@ FormatException or } public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, string oldValue) + { + PushValue(callback, column, value, oldValue.AsMemory()); + } + + public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, ReadOnlyMemory oldValue) { if (column == 1) { try { - var formatInfo = _timeFormatDeterminer.DetermineTimeFormatInfo(oldValue.AsSpan()); + var formatInfo = _timeFormatDeterminer.DetermineTimeFormatInfo(oldValue.Span); if (formatInfo == null) { return; } var newDateTime = DateTime.ParseExact(value, formatInfo.TimeFormat, formatInfo.CultureInfo); - var oldDateTime = DateTime.ParseExact(oldValue, formatInfo.TimeFormat, formatInfo.CultureInfo); + var oldDateTime = DateTime.ParseExact(oldValue.Span, formatInfo.TimeFormat, formatInfo.CultureInfo); var mSecsOld = oldDateTime.Ticks / TimeSpan.TicksPerMillisecond; var mSecsNew = newDateTime.Ticks / TimeSpan.TicksPerMillisecond; _timeOffset = (int)(mSecsNew - mSecsOld); @@ -221,8 +234,10 @@ public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, st /// /// The name of the log file to evaluate. Cannot be null. /// A collection of log lines to analyze for timestamp patterns. Cannot be null. - /// A value indicating the priority for processing the specified log file. Returns Priority.WellSupport if the - /// majority of log lines contain recognizable timestamps; otherwise, returns Priority.NotSupport. + /// + /// A value indicating the priority for processing the specified log file. Returns Priority.WellSupport if the + /// majority of log lines contain recognizable timestamps; otherwise, returns Priority.NotSupport. + /// public Priority GetPriority (string fileName, IEnumerable samples) { ArgumentNullException.ThrowIfNull(samples, nameof(samples)); diff --git a/src/LogExpert.Core/Classes/Log/BatchedProgressReporter.cs b/src/LogExpert.Core/Classes/Log/BatchedProgressReporter.cs deleted file mode 100644 index 03d711bb..00000000 --- a/src/LogExpert.Core/Classes/Log/BatchedProgressReporter.cs +++ /dev/null @@ -1,106 +0,0 @@ -using System.Collections.Concurrent; - -using LogExpert.Core.EventArguments; - -namespace LogExpert.Core.Classes.Log; - -/// -/// Batches progress updates to reduce UI thread marshalling overhead. -/// Collects updates in a thread-safe queue and processes them on a timer. -/// -//TODO Refactor -public sealed class BatchedProgressReporter : IDisposable -{ - private readonly ConcurrentQueue _progressQueue = new(); - private readonly Timer _timer; - private readonly Action _progressCallback; - private readonly int _updateIntervalMs; - private bool _disposed; - - /// - /// Creates a new batched progress reporter. - /// - /// Callback to invoke with latest progress - /// Update interval in milliseconds (default: 100ms) - public BatchedProgressReporter (Action progressCallback, int updateIntervalMs = 100) - { - _progressCallback = progressCallback ?? throw new ArgumentNullException(nameof(progressCallback)); - _updateIntervalMs = updateIntervalMs; - - // Start timer - _timer = new Timer(ProcessQueue, null, updateIntervalMs, updateIntervalMs); - } - - /// - /// Reports progress (thread-safe, non-blocking) - /// - public void ReportProgress (LoadFileEventArgs args) - { - if (_disposed) - { - return; - } - - // Only keep the latest update - discard old ones - _progressQueue.Enqueue(args); - - // Keep queue size bounded (max 10 items) - while (_progressQueue.Count > 10) - { - _ = _progressQueue.TryDequeue(out _); - } - } - - /// - /// Flushes any pending updates immediately - /// - public void Flush () - { - ProcessQueue(null); - } - - private void ProcessQueue (object state) - { - if (_disposed) - { - return; - } - - // Get only the LATEST update (discard intermediate ones) - LoadFileEventArgs latestUpdate = null; - while (_progressQueue.TryDequeue(out var update)) - { - latestUpdate = update; - } - - // Invoke callback with latest update - if (latestUpdate != null) - { - try - { - _progressCallback(latestUpdate); - } - catch (Exception ex) - { - // Log but don't crash - System.Diagnostics.Debug.WriteLine($"Error in progress callback: {ex.Message}"); - } - } - } - - public void Dispose () - { - if (_disposed) - { - return; - } - - _disposed = true; - - Flush(); - _timer?.Dispose(); - - // Clear queue - _progressQueue.Clear(); - } -} \ No newline at end of file diff --git a/src/LogExpert.Core/Classes/Log/Buffers/BufferIndex.cs b/src/LogExpert.Core/Classes/Log/Buffers/BufferIndex.cs new file mode 100644 index 00000000..d70cbb7d --- /dev/null +++ b/src/LogExpert.Core/Classes/Log/Buffers/BufferIndex.cs @@ -0,0 +1,656 @@ +using System.Collections.Concurrent; +using System.Diagnostics; +using System.Globalization; + +using NLog; + +namespace LogExpert.Core.Classes.Log.Buffers; + +/* + * !IMPORTANT + * Before and after changes are made run the BufferIndexBenchmarks for a baseline, so no performance regression is introduced + * If changes are made to this class, please also review BufferIndexSnapshot and BufferShiftTest to ensure consistency and correctness. + */ + +/// +/// Thread-safe index that maps line numbers to instances with LRU eviction. This is the hot +/// path — every GetLogLine call goes through here. Has zero file-I/O dependencies. Constructable with only integers for +/// benchmarking. +/// +public sealed class BufferIndex : IDisposable +{ + private readonly int _maxBuffers; + private readonly int _maxLinesPerBuffer; + private readonly ReaderWriterLockSlim _lock = new(LockRecursionPolicy.SupportsRecursion); + private readonly SortedList _bufferList = []; + private readonly ConcurrentDictionary _lruCacheDict; + private readonly ThreadLocal _lastBufferIndex = new(() => -1); + + private static readonly Logger _logger = LogManager.GetCurrentClassLogger(); + + private volatile bool _isLineCountDirty = true; + private int _cachedLineCount; + + public BufferIndex (int maxBuffers, int maxLinesPerBuffer) + { + _maxBuffers = maxBuffers; + _maxLinesPerBuffer = maxLinesPerBuffer; + _lruCacheDict = new(Environment.ProcessorCount, maxBuffers + 1); + } + + #region Hot Path Lookup + + /// + /// 4-layer lookup. Caller must hold at least a read lock. Returns false if lineNum is out of range or the index is + /// empty. + /// + public LogBufferEntry TryFindBuffer (int lineNum) + { + return TryFindBufferWithIndex(lineNum); + } + + /// + /// Core buffer lookup returning both buffer and index position. The caller MUST already hold a read, + /// upgradeable-read, or write lock. + /// + internal LogBufferEntry GetBufferForLineWithIndex (int lineNum) + { + return TryFindBufferWithIndex(lineNum); + } + + private LogBufferEntry TryFindBufferWithIndex (int lineNum) + { +#if DEBUG + Util.AssertTrue(_lock.IsReadLockHeld || _lock.IsUpgradeableReadLockHeld || _lock.IsWriteLockHeld, "No lock held for buffer list in TryFindBufferWithIndex"); + long startTime = Environment.TickCount; +#endif + var arr = _bufferList.Values; + var count = arr.Count; + + if (count == 0) + { + return new LogBufferEntry(null, -1, false); + } + + // Layer 0: Last buffer cache — O(1) for sequential access + var lastIdx = _lastBufferIndex.Value; + if (lastIdx >= 0 && lastIdx < count) + { + var buf = arr[lastIdx]; + if ((uint)(lineNum - buf.StartLine) < (uint)buf.LineCount) + { + //dont UpdateLRUCache, the cache has not changed in layer 0 + return new LogBufferEntry(buf, lastIdx, true); + } + + // Layer 1: Adjacent buffer prediction — O(1) for buffer boundary crossings + if (lastIdx + 1 < count) + { + var next = arr[lastIdx + 1]; + if ((uint)(lineNum - next.StartLine) < (uint)next.LineCount) + { + _lastBufferIndex.Value = lastIdx + 1; + UpdateLru(next); + return new LogBufferEntry(next, lastIdx + 1, true); + } + } + + if (lastIdx - 1 >= 0) + { + var prev = arr[lastIdx - 1]; + if ((uint)(lineNum - prev.StartLine) < (uint)prev.LineCount) + { + _lastBufferIndex.Value = lastIdx - 1; + UpdateLru(prev); + return new LogBufferEntry(prev, lastIdx - 1, true); + } + } + } + + // Layer 2: Direct mapping guess — O(1) speculative for uniform buffers + var guess = lineNum / _maxLinesPerBuffer; + if ((uint)guess < (uint)count) + { + var buf = arr[guess]; + if ((uint)(lineNum - buf.StartLine) < (uint)buf.LineCount) + { + _lastBufferIndex.Value = guess; + UpdateLru(buf); + return new LogBufferEntry(buf, guess, true); + } + } + + // Layer 3: Branchless binary search with power-of-two strides + var step = HighestPowerOfTwo(count); + var idx = (arr[step - 1].StartLine <= lineNum) ? count - step : 0; + + for (step >>= 1; step > 0; step >>= 1) + { + var probe = idx + step; + if (probe < count && arr[probe].StartLine <= lineNum) + { + idx = probe; + } + } + + // idx is now the buffer index — verify bounds + if (idx < count) + { + var buf = arr[idx]; + if ((uint)(lineNum - buf.StartLine) < (uint)buf.LineCount) + { + _lastBufferIndex.Value = idx; + UpdateLru(buf); + return new LogBufferEntry(buf, idx, true); + } + } +#if DEBUG + long endTime = Environment.TickCount; + _logger.Debug($"TryFindBufferWithIndex({lineNum}) duration: {endTime - startTime} ms."); +#endif + return new LogBufferEntry(null, -1, false); + } + + #endregion + + #region Navigation: multi-file traversal + + /// + /// Finds the start line of the next file segment after . Caller must hold at least a read + /// lock. + /// + public (bool Found, int StartLine) TryGetNextFileStartLine (int lineNum) + { + var result = -1; + + var foundBufferEntry = TryFindBufferWithIndex(lineNum); + if (!foundBufferEntry.Found) + { + return (foundBufferEntry.Found, result); + } + + for (var i = foundBufferEntry.Index; i < _bufferList.Values.Count; ++i) + { + if (_bufferList.Values[i].FileInfo != foundBufferEntry.Buffer.FileInfo) + { + result = _bufferList.Values[i].StartLine; + break; + } + } + + return (result != -1, result); + } + + /// + /// Finds the start line of the previous file segment before . Caller must hold at least a + /// read lock. + /// + public (bool Found, int StartLine) TryGetPrevFileStartLine (int lineNum) + { + var result = -1; + + var foundBufferEntry = TryFindBufferWithIndex(lineNum); + + if (!foundBufferEntry.Found) + { + return (foundBufferEntry.Found, result); + } + + if (foundBufferEntry.Buffer != null && foundBufferEntry.Index != -1) + { + for (var i = foundBufferEntry.Index; i >= 0; --i) + { + if (_bufferList.Values[i].FileInfo != foundBufferEntry.Buffer.FileInfo) + { + result = _bufferList.Values[i].StartLine + _bufferList.Values[i].LineCount; + break; + } + } + } + + return (result != -1, result); + } + + /// + /// Finds the first buffer belonging to the same file as . Caller must hold at least a + /// read lock. + /// + public LogBuffer? GetFirstBufferForFile (LogBuffer logBuffer, int index) + { + //maybe not necessary + ArgumentNullException.ThrowIfNull(logBuffer, "GetFirstBufferForFile not possible: Buffer is NULL"); + + if (index == -1) + { + return null; + } + + var info = logBuffer.FileInfo; + + var resultBuffer = logBuffer; + while (true) + { + index--; + if (index < 0 || _bufferList.Values[index].FileInfo != info) + { + break; + } + + resultBuffer = _bufferList.Values[index]; + } + + return resultBuffer; + } + + #endregion + + #region Mutation — called during reads and rollover + + /// + /// Adds a buffer to the index and updates LRU tracking. Caller must hold a write lock. + /// + public void Add (LogBuffer buffer) + { +#if DEBUG + _logger.Debug(CultureInfo.InvariantCulture, "AddBufferToList(): {0}/{1}/{2}", buffer.StartLine, buffer.LineCount, buffer.FileInfo.FullName); +#endif + _bufferList[buffer.StartLine] = buffer; + UpdateLru(buffer); + _isLineCountDirty = true; + } + + /// + /// Removes a buffer by its start line key and LRU entry. Caller must hold a write lock. + /// + public bool Remove (LogBuffer buffer) + { + ArgumentNullException.ThrowIfNull(buffer, "Remove not possible: Buffer is NULL"); + + Debug.Assert(_lock.IsWriteLockHeld, "No writer lock for buffer list"); + _ = _lruCacheDict.TryRemove(buffer.StartLine, out _); + _isLineCountDirty = true; + return _bufferList.Remove(buffer.StartLine); + } + + /// + /// Atomically updates a buffer's start line in both the index and LRU cache. Used by ShiftBuffers during rollover. + /// Caller must hold a write lock. + /// + public void UpdateStartLine (LogBuffer buffer, int newStartLine) + { + var hadCache = _lruCacheDict.TryRemove(buffer.StartLine, out var cacheEntry); + + _ = _bufferList.Remove(buffer.StartLine); + buffer.StartLine = newStartLine; + _bufferList[newStartLine] = buffer; + + if (hadCache) + { + _ = _lruCacheDict.TryAdd(buffer.StartLine, cacheEntry); + } + + _isLineCountDirty = true; + } + + /// + /// Clears all buffers and LRU entries. Does NOT dispose buffer content. Caller must hold a write lock. + /// + public void Clear () + { + _bufferList.Clear(); + _lruCacheDict.Clear(); + ResetThreadLocalCache(); + _isLineCountDirty = true; + } + + #endregion + + #region LRU eviction + + /// + /// Removes least-recently-used entries when cache exceeds max size. Evicts content but preserves metadata so + /// buffers remain findable for re-read. Does NOT acquire _lock — only touches _lruCache (ConcurrentDictionary) and + /// individual buffer SpinLocks. + /// + public void EvictLeastRecentlyUsed () + { +#if DEBUG + long startTime = Environment.TickCount; +#endif + _logger.Debug(CultureInfo.InvariantCulture, "Starting garbage collection"); + var threshold = 10; + + if (_lruCacheDict.Count - (_maxBuffers + threshold) > 0) + { + var diff = _lruCacheDict.Count - _maxBuffers; +#if DEBUG + if (diff > 0) + { + _logger.Info(CultureInfo.InvariantCulture, "Removing {0} entries from LRU cache", diff); + } +#endif + // Snapshot values and sort by timestamp (ascending = least recently used first) + var entries = _lruCacheDict.ToArray(); + Array.Sort(entries, static (a, b) => a.Value.LastUseTimeStamp.CompareTo(b.Value.LastUseTimeStamp)); + + for (var i = 0; i < diff && i < entries.Length; ++i) + { + var kvp = entries[i]; + if (_lruCacheDict.TryRemove(kvp.Key, out var removed)) + { + // Skip pinned buffers — the UI is actively displaying their content. + // Re-add to LRU so they'll be reconsidered in a future eviction pass. + if (removed.LogBuffer.IsPinned) + { + _lruCacheDict.TryAdd(kvp.Key, removed); + continue; + } + + var lockTaken = false; + try + { + removed.LogBuffer.AcquireContentLock(ref lockTaken); + // Evict content but preserve metadata (LineCount, StartLine, etc.) + // so the buffer remains findable in _bufferList lookups. + // Do NOT return to pool — the buffer is still referenced by _bufferList. + removed.LogBuffer.EvictContent(); + } + finally + { + if (lockTaken) + { + removed.LogBuffer.ReleaseContentLock(); + } + } + } + } + } + +#if DEBUG + if (_lruCacheDict.Count - (_maxBuffers + threshold) > 0) + { + long endTime = Environment.TickCount; + _logger.Info(CultureInfo.InvariantCulture, "Garbage collector time: " + (endTime - startTime) + " ms."); + } +#endif + } + + /// + /// Pins all buffers that cover the specified line range. Returns a + /// that unpins them on dispose. Caller must hold at least a read lock. + /// + public PinHandle PinRange (int startLine, int endLine) + { + var pinned = new List(); + var line = startLine; + + while (line <= endLine) + { + var entry = TryFindBuffer(line); + if (!entry.Found || entry.Buffer is null) + { + break; + } + + entry.Buffer.Pin(); + pinned.Add(entry.Buffer); + + // Jump to next buffer's start line to avoid redundant lookups + line = entry.Buffer.StartLine + entry.Buffer.LineCount; + } + + return new PinHandle(pinned); + } + + /// + /// Atomically clears the index and returns all LRU-tracked buffers to the pool. Clears the index FIRST under the + /// caller's write lock, THEN returns buffers to pool. This prevents a race where concurrent readers find buffers + /// that have been returned to the pool. Caller must hold a write lock. + /// + public void ClearLru (LogBufferPool pool) + { + _logger.Info(CultureInfo.InvariantCulture, "Clearing LRU cache."); + + // 1. Collect buffer references before clearing + var toReturn = new List(_lruCacheDict.Count); + foreach (var entry in _lruCacheDict.Values) + { + toReturn.Add(entry.LogBuffer); + } + + // 2. Clear index FIRST — no concurrent reader can find these after this + _bufferList.Clear(); + _lruCacheDict.Clear(); + _isLineCountDirty = true; + ResetThreadLocalCache(); + + // 3. Now safe to return to pool + foreach (var entry in toReturn) + { + var lockTaken = false; + try + { + entry.AcquireContentLock(ref lockTaken); + pool.Return(entry); + } + finally + { + if (lockTaken) + { + entry.ReleaseContentLock(); + } + } + } + + _logger.Info(CultureInfo.InvariantCulture, "Clearing done."); + } + + #endregion + + /// + /// Gets the number of buffers. + /// + public int BufferCount => _bufferList.Count; + + /// + /// Returns the buffer at the specified positional index. Caller must hold at least a read lock. + /// + public LogBuffer GetBufferAt (int index) => _bufferList.GetValueAtIndex(index); + + /// + /// Returns the last buffer in the index (highest start line). Caller must hold at least a read lock. + /// + public LogBuffer GetLastBuffer () => _bufferList.GetValueAtIndex(_bufferList.Count - 1); + + /// + /// Returns an enumerable collection of all log buffers managed by the current instance. + /// + /// + /// An containing each in the collection. The + /// enumeration reflects the current state of the buffers at the time of the call. + /// + public IEnumerable EnumerateBuffers () { return [.. _bufferList.Values]; } + + /// + /// Total lines across all buffers. Recalculated on demand when dirty. Caller must hold at least a read lock. + /// + public int TotalLineCount + { + get + { + if (_isLineCountDirty) + { + var total = 0; + foreach (var buffer in _bufferList.Values) + { + total += buffer.LineCount; + } + + _cachedLineCount = total; + _isLineCountDirty = false; + } + + return _cachedLineCount; + } + } + + public void MarkLineCountDirty () => _isLineCountDirty = true; + + /// + /// Gets the number of items currently stored in the least recently used (LRU) cache. + /// + public int LruCacheCount => _lruCacheDict.Count; + + #region Lock management — using-scoped only + + public ReadLockScope AcquireReadLock () => new(_lock); + + public WriteLockScope AcquireWriteLock () => new(_lock); + + public UpgradeableReadLockScope AcquireUpgradeableReadLock () => new(_lock); + + #endregion + + #region Diagnostics + + /// + /// Creates an immutable point-in-time capture of the index state. Acquires its own read lock internally. + /// + public BufferIndexSnapshot CreateSnapshot () + { + using var _ = AcquireReadLock(); + + var buffers = new List(_bufferList.Count); + + foreach (var b in _bufferList.Values) + { + buffers.Add(new BufferIndexSnapshot.BufferInfo + ( + b.StartLine, + b.LineCount, + b.StartPos, + b.Size, + b.IsDisposed, + b.FileInfo.FullName + )); + } + + return new BufferIndexSnapshot + { + BufferCount = _bufferList.Count, + TotalLineCount = TotalLineCount, + LruCacheCount = _lruCacheDict.Count, + Buffers = buffers + }; + } + + #endregion + + #region Internal Helpers + + public void ResetThreadLocalCache () => _lastBufferIndex.Value = -1; + + private void UpdateLru (LogBuffer logBuffer) + { + var cacheEntry = _lruCacheDict.GetOrAdd( + logBuffer.StartLine, + static (_, buf) => new LogBufferCacheEntry { LogBuffer = buf }, + logBuffer); + + cacheEntry.Touch(); + } + + private static int HighestPowerOfTwo (int n) => 1 << (31 - int.LeadingZeroCount(n)); + + public void Dispose () + { + _lastBufferIndex.Dispose(); + _lock.Dispose(); + } + + #endregion +} + +#region Lock scope structs + +public readonly ref struct ReadLockScope +{ + private readonly ReaderWriterLockSlim _lock; + + public ReadLockScope (ReaderWriterLockSlim rwLock) + { + _lock = rwLock; + if (!_lock.TryEnterReadLock(TimeSpan.FromSeconds(10))) + { + //_logger.Warn("Reader lock wait timed out, forcing entry"); + _lock.EnterReadLock(); + } + } + + public void Dispose () => _lock.ExitReadLock(); + +} + +public readonly ref struct WriteLockScope +{ + private readonly ReaderWriterLockSlim _lock; + + public WriteLockScope (ReaderWriterLockSlim rwLock) + { + _lock = rwLock; + if (!_lock.TryEnterWriteLock(TimeSpan.FromSeconds(10))) + { + //_logger.Warn("Writer lock wait timed out, forcing entry"); + _lock.EnterWriteLock(); + } + } + + public void Dispose () => _lock.ExitWriteLock(); +} + +public readonly ref struct UpgradeableReadLockScope +{ + private readonly ReaderWriterLockSlim _lock; + + public UpgradeableReadLockScope (ReaderWriterLockSlim rwLock) + { + _lock = rwLock; + if (!_lock.TryEnterUpgradeableReadLock(TimeSpan.FromSeconds(10))) + { + //_logger.Warn("Upgradeable read lock timed out, forcing entry"); + _lock.EnterUpgradeableReadLock(); + } + } + + public WriteLockUpgradeScope UpgradeToWrite () => new(_lock); + + public void Dispose () => _lock.ExitUpgradeableReadLock(); +} + +public readonly ref struct WriteLockUpgradeScope +{ + private readonly ReaderWriterLockSlim _lock; + + public WriteLockUpgradeScope (ReaderWriterLockSlim rwls) + { + _lock = rwls; + if (!_lock.TryEnterWriteLock(TimeSpan.FromSeconds(10))) + { + //_logger.Warn("Writer lock upgrade timed out, forcing entry"); + _lock.EnterWriteLock(); + } + } + + public void Dispose () => _lock.ExitWriteLock(); +} + +#endregion + +public readonly struct LogBufferEntry (LogBuffer? buffer, int index, bool found) +{ + public LogBuffer? Buffer { get; } = buffer; + + public int Index { get; } = index; + + public bool Found { get; } = found; +} \ No newline at end of file diff --git a/src/LogExpert.Core/Classes/Log/Buffers/BufferIndexSnapshot.cs b/src/LogExpert.Core/Classes/Log/Buffers/BufferIndexSnapshot.cs new file mode 100644 index 00000000..5b54390b --- /dev/null +++ b/src/LogExpert.Core/Classes/Log/Buffers/BufferIndexSnapshot.cs @@ -0,0 +1,24 @@ +namespace LogExpert.Core.Classes.Log.Buffers; + +/// +/// Immutable point-in-time capture of state. +/// Taken under a single read lock, safe to inspect afterward without locks. +/// +public sealed class BufferIndexSnapshot +{ + public int BufferCount { get; init; } + public int TotalLineCount { get; init; } + public int LruCacheCount { get; init; } + public IReadOnlyList Buffers { get; init; } = []; + + public sealed record BufferInfo ( + int StartLine, + int LineCount, + long StartPos, + long Size, + bool IsDisposed, + string FileName); + + public override string ToString () => + $"Buffers={BufferCount}, Lines={TotalLineCount}, LRU={LruCacheCount}"; +} \ No newline at end of file diff --git a/src/LogExpert.Core/Classes/Log/Buffers/CharBlockAllocator.cs b/src/LogExpert.Core/Classes/Log/Buffers/CharBlockAllocator.cs new file mode 100644 index 00000000..50061f94 --- /dev/null +++ b/src/LogExpert.Core/Classes/Log/Buffers/CharBlockAllocator.cs @@ -0,0 +1,136 @@ +using System.Buffers; + +namespace LogExpert.Core.Classes.Log.Buffers; + +/// +/// Blocks are rented from and returned when the owning is +/// evicted. The pinning mechanism (Phase 1/2) ensures that buffers whose content is still displayed by the UI are +/// exempt from eviction, preventing use-after-return corruption. +/// This class is NOT thread-safe. Each reader/fill operation should use its own instance. +/// +public sealed class CharBlockAllocator : IDisposable +{ + private const int DEFAULT_BLOCK_SIZE = 32_768; // 64 KB (32K chars × 2 bytes), stays under 85 KB LOH threshold + + private readonly int _blockSize; + private List _blocks = []; + private readonly List _oversizedBlocks = []; + private char[] _currentBlock; + private int _currentOffset; + private bool _disposed; + + public CharBlockAllocator (int blockSize = DEFAULT_BLOCK_SIZE) + { + _blockSize = blockSize; + _currentBlock = ArrayPool.Shared.Rent(_blockSize); + _blocks.Add(_currentBlock); + _currentOffset = 0; + } + + /// + /// Gets the number of normal (fixed-size) blocks currently rented from the pool. + /// + public int BlockCount => _blocks.Count; + + /// + /// Gets the number of oversized (standalone) blocks currently rented from the pool. + /// Useful for diagnostics — a high count indicates pathological line lengths. + /// + public int OversizedBlockCount => _oversizedBlocks.Count; + + /// + /// Allocates a region of the specified length from the current block. + /// If the current block has insufficient space, a new block is rented. + /// Lines longer than the block size receive a standalone rental tracked separately. + /// + public Memory Rent (int length) + { + ObjectDisposedException.ThrowIf(_disposed, this); + + if (length <= 0) + { + return Memory.Empty; + } + + // Oversized line: give it its own array, tracked separately + if (length > _blockSize) + { + var oversized = new char[length]; + _oversizedBlocks.Add(oversized); + return oversized.AsMemory(0, length); + } + + // Current block has space + if (_currentOffset + length <= _currentBlock.Length) + { + var memory = _currentBlock.AsMemory(_currentOffset, length); + _currentOffset += length; + return memory; + } + + // Need a new block + _currentBlock = ArrayPool.Shared.Rent(_blockSize); + _blocks.Add(_currentBlock); + _currentOffset = length; + return _currentBlock.AsMemory(0, length); + } + + /// + /// Detaches and returns the list of all blocks (normal + oversized). After this call, + /// the allocator no longer owns those blocks — the caller (LogBuffer) holds them + /// until GC collects them after all slices are released. + /// + public List DetachBlocks () + { + ObjectDisposedException.ThrowIf(_disposed, this); + + // Merge oversized blocks into the main list so the caller owns everything + if (_oversizedBlocks.Count > 0) + { + _blocks.AddRange(_oversizedBlocks); + _oversizedBlocks.Clear(); + } + + // Swap the list — O(1), no copy. Caller owns the old list. + var blocks = _blocks; + _currentBlock = ArrayPool.Shared.Rent(_blockSize); + _blocks = [_currentBlock]; + _currentOffset = 0; + return blocks; + } + + /// + /// Returns all blocks to . + /// Safe to call only when no slices reference these blocks + /// (i.e., after DetachBlocks transferred ownership to LogBuffer, and the reader is being disposed). + /// + public void ReturnAll () + { + foreach (var block in _blocks) + { + ArrayPool.Shared.Return(block); + } + + _blocks.Clear(); + + foreach (var block in _oversizedBlocks) + { + ArrayPool.Shared.Return(block); + } + + _oversizedBlocks.Clear(); + _currentBlock = null!; + _currentOffset = 0; + } + + public void Dispose () + { + if (_disposed) + { + return; + } + + ReturnAll(); + _disposed = true; + } +} \ No newline at end of file diff --git a/src/LogExpert.Core/Classes/Log/Buffers/LogBuffer.cs b/src/LogExpert.Core/Classes/Log/Buffers/LogBuffer.cs new file mode 100644 index 00000000..020752fb --- /dev/null +++ b/src/LogExpert.Core/Classes/Log/Buffers/LogBuffer.cs @@ -0,0 +1,346 @@ +using System.Buffers; + +using ColumnizerLib; + +using NLog; + +namespace LogExpert.Core.Classes.Log.Buffers; + +public class LogBuffer +{ + #region Fields + + private SpinLock _contentLock = new(enableThreadOwnerTracking: false); + private static readonly Logger _logger = LogManager.GetCurrentClassLogger(); + +#if DEBUG + private readonly List _filePositions; // file position for every line +#endif + + private LogLine[] _lineArray; + private int _lineArrayLength; // capacity of the rented array + private List _charBlocks; + + private int _pinCount; + + private int MAX_LINES = 500; + + #endregion + + #region cTor + + // Don't use a primary constructor here: field initializers (like MAX_LINES) run before primary constructor parameters are assigned, + // so MAX_LINES would always be set to its default value before the constructor body can assign it. Use a regular constructor instead. + public LogBuffer (ILogFileInfo fileInfo, int maxLines) + { + FileInfo = fileInfo; + MAX_LINES = maxLines; + _lineArray = ArrayPool.Shared.Rent(maxLines); + _lineArrayLength = _lineArray.Length; +#if DEBUG + _filePositions = new(MAX_LINES); +#endif + } + + #endregion + + #region Properties + + /// + /// Returns true if any component has pinned this buffer to prevent eviction. + /// + public bool IsPinned => Volatile.Read(ref _pinCount) > 0; + + public long StartPos { set; get; } + + public long Size + { + set + { + field = value; +#if DEBUG + if (_filePositions.Count > 0) + { + if (field < _filePositions[^1] - StartPos) + { + _logger.Error("### LogBuffer: LogBuffer overall Size must be greater than last line file position!"); + } + } +#endif + } + get; + } + + public int EndLine => StartLine + LineCount; + + public int StartLine { set; get; } + + public int LineCount { get; private set; } + + public bool IsDisposed { get; private set; } + + public ILogFileInfo FileInfo { get; set; } + + public int DroppedLinesCount { get; set; } + + public int PrevBuffersDroppedLinesSum { get; set; } + + #endregion + + #region Public methods + + /// + /// Increments the pin count. While pinned, the buffer will not be evicted by the LRU garbage collector. Each call + /// to Pin() must be balanced by a call to Unpin(). + /// + public void Pin () + { + _ = Interlocked.Increment(ref _pinCount); + } + + /// + /// Decrements the pin count. When the count reaches zero, the buffer becomes eligible for eviction. + /// + public void Unpin () + { +#if DEBUG + var newCount = Interlocked.Decrement(ref _pinCount); + if (newCount < 0) + { + _logger.Warn("Unpin underflow: _pinCount went to {0}. Unbalanced Pin/Unpin calls.", newCount); + } +#else + Interlocked.Decrement(ref _pinCount); +#endif + } + + /// + /// Adds a log line to the internal collection at the specified file position. + /// + /// + /// If the internal collection has reached its maximum capacity, the log line is not added. In debug builds, an + /// error is logged when this occurs. + /// + /// The log line to add to the collection. + /// The file position associated with the log line. + public void AddLine (LogLine lineMemory, long filePos) + { + if (LineCount < _lineArrayLength) + { + _lineArray[LineCount] = lineMemory; + LineCount++; + } +#if DEBUG + else + { + _logger.Error("AddLine overflow: LineCount={0} >= _lineArrayLength={1}", LineCount, _lineArrayLength); + } +#endif + +#if DEBUG + _filePositions.Add(filePos); +#endif + IsDisposed = false; + } + + /// + /// Removes all log lines from the current collection, resetting its state for reuse. + /// + /// + /// After calling this method, the collection will be empty and ready to accept new log lines. Any resources + /// associated with the previous log lines are released. This method is typically used to clear the log data before + /// loading new content or starting a new logging session. + /// + public void ClearLines () + { + if (_lineArray == null) + { + _lineArray = ArrayPool.Shared.Rent(MAX_LINES); + _lineArrayLength = _lineArray.Length; + } + else + { + Array.Clear(_lineArray, 0, LineCount); + } + + ReturnCharBlocks(); + + LineCount = 0; +#if DEBUG + _filePositions.Clear(); +#endif + } + + /// + /// Prepares the buffer for reuse from the pool. + /// + public void Reinitialise (ILogFileInfo fileInfo, int maxLines) + { + ReturnCharBlocks(); + + FileInfo = fileInfo; + MAX_LINES = maxLines; + StartLine = 0; + StartPos = 0; + Size = 0; + LineCount = 0; + DroppedLinesCount = 0; + PrevBuffersDroppedLinesSum = 0; + IsDisposed = false; + _pinCount = 0; + _lineArray = ArrayPool.Shared.Rent(maxLines); + _lineArrayLength = _lineArray.Length; +#if DEBUG + _filePositions.Clear(); + DisposeCount = 0; +#endif + } + + /// + /// Evicts the buffer content to free memory while preserving metadata (LineCount, StartLine, StartPos, Size). The + /// buffer remains findable in buffer list lookups and can be re-read from disk when accessed. + /// + public void EvictContent () + { + if (_lineArray != null) + { + Array.Clear(_lineArray, 0, LineCount); + ArrayPool.Shared.Return(_lineArray); + _lineArray = null; + } + + ReturnCharBlocks(); + + //! Do NOT zero LineCount — it is needed for buffer lookup in GetBufferForLineWithIndex. + //! Do NOT zero StartLine, StartPos, Size — they are needed for re-reading from disk. + IsDisposed = true; +#if DEBUG + DisposeCount++; +#endif + } + + /// + /// Fully disposes the buffer content and resets all metadata. Used when the buffer is being returned to the pool or + /// completely removed from the buffer list. + /// + public void DisposeContent () + { + if (_lineArray != null) + { + Array.Clear(_lineArray, 0, LineCount); + ArrayPool.Shared.Return(_lineArray); + _lineArray = null; + LineCount = 0; + } + + ReturnCharBlocks(); + + IsDisposed = true; +#if DEBUG + DisposeCount++; +#endif + } + + /// + /// Retrieves the log line at the specified index within the current memory block. + /// + /// + /// The zero-based index of the log line to retrieve. Must be greater than or equal to 0 and less than the total + /// number of lines. + /// + /// + /// The at the specified index if it exists; otherwise, . + /// + public LogLine? GetLineMemoryOfBlock (int num) + { + return num < LineCount && num >= 0 + ? _lineArray[num] + : null; + } + + /// + /// Acquires the content lock. The caller MUST call in a finally block. + /// + public void AcquireContentLock (ref bool lockTaken) + { + _contentLock.Enter(ref lockTaken); + } + + /// + /// Releases the content lock previously acquired via . + /// + public void ReleaseContentLock () + { + _contentLock.Exit(useMemoryBarrier: false); + } + + /// + /// Attaches pooled char[] blocks that back the ReadOnlyMemory in this buffer's LogLine entries. These blocks will + /// be returned to ArrayPool when the buffer is evicted or disposed. New blocks are MERGED with existing ones — + /// never replace — because the buffer's existing LogLine entries still reference the old blocks (e.g., during tail + /// mode where multiple read sessions append lines to the same buffer). + /// + public void AttachCharBlocks (List blocks) + { + if (_charBlocks is null) + { + _charBlocks = blocks; + } + else + { + _charBlocks.AddRange(blocks); + } + } + + #endregion + +#if DEBUG + + public long DisposeCount { get; private set; } + + public long GetFilePosForLineOfBlock (int line) + { + return line >= 0 && line < _filePositions.Count + ? _filePositions[line] + : -1; + } + +#endif + + #region Private Methods + + /// + /// Releases references to the character block buffers used by this instance, allowing them to be garbage collected + /// when no longer in use. + /// + /// + /// If the buffer is pinned, this method only drops the reference without returning the blocks to the array pool, as + /// external consumers may still hold references. This helps prevent premature reuse of buffers that may still be + /// accessed elsewhere. + /// + private void ReturnCharBlocks () + { + if (_charBlocks is null) + { + return; + } + + if (IsPinned) + { + // Buffer is pinned — UI still holds ReadOnlyMemory slices into these blocks. + // Don't return to ArrayPool; just drop the reference. GC will collect them + // once all UI references (ColumnCache, DataGridView) are released. + _charBlocks = null; + return; + } + + foreach (var block in _charBlocks) + { + ArrayPool.Shared.Return(block); + } + + _charBlocks = null; + } + + #endregion +} \ No newline at end of file diff --git a/src/LogExpert.Core/Classes/Log/LogBufferCacheEntry.cs b/src/LogExpert.Core/Classes/Log/Buffers/LogBufferCacheEntry.cs similarity index 91% rename from src/LogExpert.Core/Classes/Log/LogBufferCacheEntry.cs rename to src/LogExpert.Core/Classes/Log/Buffers/LogBufferCacheEntry.cs index b983cfed..7ce51e47 100644 --- a/src/LogExpert.Core/Classes/Log/LogBufferCacheEntry.cs +++ b/src/LogExpert.Core/Classes/Log/Buffers/LogBufferCacheEntry.cs @@ -1,4 +1,4 @@ -namespace LogExpert.Core.Classes.Log; +namespace LogExpert.Core.Classes.Log.Buffers; public class LogBufferCacheEntry { diff --git a/src/LogExpert.Core/Classes/Log/LogBufferPool.cs b/src/LogExpert.Core/Classes/Log/Buffers/LogBufferPool.cs similarity index 95% rename from src/LogExpert.Core/Classes/Log/LogBufferPool.cs rename to src/LogExpert.Core/Classes/Log/Buffers/LogBufferPool.cs index b7bbed59..fb825902 100644 --- a/src/LogExpert.Core/Classes/Log/LogBufferPool.cs +++ b/src/LogExpert.Core/Classes/Log/Buffers/LogBufferPool.cs @@ -2,7 +2,7 @@ using ColumnizerLib; -namespace LogExpert.Core.Classes.Log; +namespace LogExpert.Core.Classes.Log.Buffers; public sealed class LogBufferPool (int maxSize) { diff --git a/src/LogExpert.Core/Classes/Log/Buffers/PinHandle.cs b/src/LogExpert.Core/Classes/Log/Buffers/PinHandle.cs new file mode 100644 index 00000000..ac6d659b --- /dev/null +++ b/src/LogExpert.Core/Classes/Log/Buffers/PinHandle.cs @@ -0,0 +1,34 @@ +namespace LogExpert.Core.Classes.Log.Buffers; + +/// +/// Tracks a set of pinned instances. Disposing the handle +/// unpins all held buffers, making them eligible for LRU eviction. +/// +public sealed class PinHandle : IDisposable +{ + private List? _pinnedBuffers; + + internal PinHandle (List pinnedBuffers) + { + _pinnedBuffers = pinnedBuffers; + } + + /// + /// Gets the number of buffers currently pinned by this handle. + /// + public int Count => _pinnedBuffers?.Count ?? 0; + + public void Dispose () + { + var buffers = Interlocked.Exchange(ref _pinnedBuffers, null); + if (buffers is null) + { + return; + } + + foreach (var buffer in buffers) + { + buffer.Unpin(); + } + } +} \ No newline at end of file diff --git a/src/LogExpert.Core/Classes/Log/CastingPipelineBuilder.cs b/src/LogExpert.Core/Classes/Log/CastingPipelineBuilder.cs new file mode 100644 index 00000000..e0a92007 --- /dev/null +++ b/src/LogExpert.Core/Classes/Log/CastingPipelineBuilder.cs @@ -0,0 +1,67 @@ +using System.Collections.Concurrent; + +using LogExpert.Core.Interfaces; + +namespace LogExpert.Core.Classes.Log; + +public class CastingPipelineBuilder : IPipeline +{ + private readonly List> _pipelineSteps = []; + private BlockingCollection[] _buffers; + + + public event Action Finished; + + public void AddStep (Func stepFunc) + { + _pipelineSteps.Add(stepFunc); + } + + public void Execute (object input) + { + BlockingCollection first = _buffers[0]; + first.Add(input); + } + + public void Complete () + { + if (_buffers.Length > 0) + { + _buffers[0].CompleteAdding(); + } + } + + public IPipeline GetPipeline () + { + //Create Buffers + _buffers = _pipelineSteps.Select(step => new BlockingCollection()).ToArray(); + + int bufferIndex = 0; + foreach (Func pipelineStep in _pipelineSteps) + { + var bufferIndexLocal = bufferIndex; //with this remains same in each thread + _ = Task.Run(() => + { + //GetConsuminEnumerable => is blocking when the collection is empty + foreach (object input in _buffers[bufferIndexLocal].GetConsumingEnumerable().Select(pipelineStep)) + { + bool isLastStep = bufferIndexLocal == _pipelineSteps.Count - 1; + if (isLastStep) + { + //BeginInvoke would be better https://stackoverflow.com/questions/1916095/how-do-i-make-an-eventhandler-run-asynchronously/16336361#16336361 + Finished?.Invoke(input); + } + else + { + BlockingCollection next = _buffers[bufferIndexLocal + 1]; + next.Add(input); + } + } + }); + + bufferIndex++; + } + + return this; + } +} diff --git a/src/LogExpert.Core/Classes/Log/LogBuffer.cs b/src/LogExpert.Core/Classes/Log/LogBuffer.cs deleted file mode 100644 index a240720b..00000000 --- a/src/LogExpert.Core/Classes/Log/LogBuffer.cs +++ /dev/null @@ -1,221 +0,0 @@ -using System.Buffers; - -using ColumnizerLib; - -using NLog; - -namespace LogExpert.Core.Classes.Log; - -public class LogBuffer -{ - #region Fields - - private SpinLock _contentLock = new(enableThreadOwnerTracking: false); - private static readonly Logger _logger = LogManager.GetCurrentClassLogger(); - -#if DEBUG - private readonly List _filePositions; // file position for every line -#endif - - private LogLine[] _lineArray; - private int _lineArrayLength; // capacity of the rented array - - private int MAX_LINES = 500; - - #endregion - - #region cTor - - // Don't use a primary constructor here: field initializers (like MAX_LINES) run before primary constructor parameters are assigned, - // so MAX_LINES would always be set to its default value before the constructor body can assign it. Use a regular constructor instead. - public LogBuffer (ILogFileInfo fileInfo, int maxLines) - { - FileInfo = fileInfo; - MAX_LINES = maxLines; - _lineArray = ArrayPool.Shared.Rent(maxLines); - _lineArrayLength = _lineArray.Length; -#if DEBUG - _filePositions = new(MAX_LINES); -#endif - } - - #endregion - - #region Properties - - public long StartPos { set; get; } - - public long Size - { - set - { - field = value; -#if DEBUG - if (_filePositions.Count > 0) - { - if (field < _filePositions[^1] - StartPos) - { - _logger.Error("LogBuffer overall Size must be greater than last line file position!"); - } - } -#endif - } - get; - } - - public int EndLine => StartLine + LineCount; - - public int StartLine { set; get; } - - public int LineCount { get; private set; } - - public bool IsDisposed { get; private set; } - - public ILogFileInfo FileInfo { get; set; } - - public int DroppedLinesCount { get; set; } - - public int PrevBuffersDroppedLinesSum { get; set; } - - #endregion - - #region Public methods - - public void AddLine (LogLine lineMemory, long filePos) - { - if (LineCount < _lineArrayLength) - { - _lineArray[LineCount] = lineMemory; - LineCount++; - } -#if DEBUG - else - { - _logger.Error("AddLine overflow: LineCount={0} >= _lineArrayLength={1}", LineCount, _lineArrayLength); - } -#endif - -#if DEBUG - _filePositions.Add(filePos); -#endif - IsDisposed = false; - } - - public void ClearLines () - { - if (_lineArray == null) - { - _lineArray = ArrayPool.Shared.Rent(MAX_LINES); - _lineArrayLength = _lineArray.Length; - } - else - { - Array.Clear(_lineArray, 0, LineCount); - } - - LineCount = 0; -#if DEBUG - _filePositions.Clear(); -#endif - } - - /// - /// Prepares the buffer for reuse from the pool. - /// - public void Reinitialise (ILogFileInfo fileInfo, int maxLines) - { - FileInfo = fileInfo; - MAX_LINES = maxLines; - StartLine = 0; - StartPos = 0; - Size = 0; - LineCount = 0; - DroppedLinesCount = 0; - PrevBuffersDroppedLinesSum = 0; - IsDisposed = false; - _lineArray = ArrayPool.Shared.Rent(maxLines); - _lineArrayLength = _lineArray.Length; -#if DEBUG - _filePositions.Clear(); - DisposeCount = 0; -#endif - } - - /// - /// Evicts the buffer content to free memory while preserving metadata (LineCount, StartLine, StartPos, Size). - /// The buffer remains findable in buffer list lookups and can be re-read from disk when accessed. - /// - public void EvictContent () - { - if (_lineArray != null) - { - Array.Clear(_lineArray, 0, LineCount); - ArrayPool.Shared.Return(_lineArray); - _lineArray = null; - } - - // Do NOT zero LineCount — it is needed for buffer lookup in GetBufferForLineWithIndex. - // Do NOT zero StartLine, StartPos, Size — they are needed for re-reading from disk. - IsDisposed = true; -#if DEBUG - DisposeCount++; -#endif - } - - /// - /// Fully disposes the buffer content and resets all metadata. Used when the buffer is being returned to the pool - /// or completely removed from the buffer list. - /// - public void DisposeContent () - { - if (_lineArray != null) - { - Array.Clear(_lineArray, 0, LineCount); - ArrayPool.Shared.Return(_lineArray); - _lineArray = null; - LineCount = 0; - } - - IsDisposed = true; -#if DEBUG - DisposeCount++; -#endif - } - - public LogLine? GetLineMemoryOfBlock (int num) - { - return num < LineCount && num >= 0 - ? _lineArray[num] - : null; - } - - /// - /// Acquires the content lock. The caller MUST call in a finally block. - /// - public void AcquireContentLock (ref bool lockTaken) - { - _contentLock.Enter(ref lockTaken); - } - - /// - /// Releases the content lock previously acquired via . - /// - public void ReleaseContentLock () - { - _contentLock.Exit(useMemoryBarrier: false); - } - - #endregion - -#if DEBUG - public long DisposeCount { get; private set; } - - public long GetFilePosForLineOfBlock (int line) - { - return line >= 0 && line < _filePositions.Count - ? _filePositions[line] - : -1; - } - -#endif -} \ No newline at end of file diff --git a/src/LogExpert.Core/Classes/Log/LogfileReader.cs b/src/LogExpert.Core/Classes/Log/LogfileReader.cs index bad4bfdf..89895817 100644 --- a/src/LogExpert.Core/Classes/Log/LogfileReader.cs +++ b/src/LogExpert.Core/Classes/Log/LogfileReader.cs @@ -1,9 +1,11 @@ -using System.Collections.Concurrent; using System.Globalization; using System.Text; using ColumnizerLib; +using LogExpert.Core.Classes.Log.Buffers; +using LogExpert.Core.Classes.Log.ProgressReporters; +using LogExpert.Core.Classes.Log.Streamreaders; using LogExpert.Core.Classes.xml; using LogExpert.Core.Entities; using LogExpert.Core.Enums; @@ -14,7 +16,7 @@ namespace LogExpert.Core.Classes.Log; -public partial class LogfileReader : IAutoLogLineMemoryColumnizerCallback, IDisposable +public partial class LogfileReader : ILogfileReader, IMultiFileNavigation, ILogfileReaderConfiguration, IBufferPinning, ILogfileReaderDiagnostics { #region Fields @@ -32,17 +34,14 @@ public partial class LogfileReader : IAutoLogLineMemoryColumnizerCallback, IDisp private readonly LogBufferPool _bufferPool; private readonly Lock _logBufferLock = new(); - private readonly ReaderWriterLockSlim _bufferListLock = new(LockRecursionPolicy.SupportsRecursion); + + private readonly ILoadProgressReporter _progressReporter; private readonly MemoryMappedFileReader _mmfReader; - private const int PROGRESS_UPDATE_INTERVAL_MS = 100; private const int WAIT_TIME = 1000; - private SortedList _bufferList; - private bool _contentDeleted; - private long _lastProgressUpdate; private long _fileLength; private Task _garbageCollectorTask; @@ -50,36 +49,81 @@ public partial class LogfileReader : IAutoLogLineMemoryColumnizerCallback, IDisp private bool _isDeleted; private IList _logFileInfoList = []; - private ConcurrentDictionary _lruCacheDict; private bool _shouldStop; private bool _disposed; private ILogFileInfo _watchedILogFileInfo; - private volatile bool _isLineCountDirty = true; - private volatile bool _isFailModeCheckCallPending; private volatile bool _isFastFailOnGetLogLine; - private readonly ThreadLocal _lastBufferIndex = new(() => -1); #endregion #region cTor /// Public constructor for single file. - public LogfileReader (string fileName, EncodingOptions encodingOptions, bool multiFile, int bufferCount, int linesPerBuffer, MultiFileOptions multiFileOptions, ReaderType readerType, IPluginRegistry pluginRegistry, int maximumLineLength) - : this([fileName], encodingOptions, multiFile, bufferCount, linesPerBuffer, multiFileOptions, readerType, pluginRegistry, maximumLineLength) + public LogfileReader ( + string fileName, + EncodingOptions encodingOptions, + bool multiFile, + int bufferCount, + int linesPerBuffer, + MultiFileOptions multiFileOptions, + ReaderType readerType, + IPluginRegistry pluginRegistry, + int maximumLineLength, + ILoadProgressReporter? progressReporter = null) + : this( + [fileName], + encodingOptions, + multiFile, + bufferCount, + linesPerBuffer, + multiFileOptions, + readerType, + pluginRegistry, + maximumLineLength, + progressReporter) { } /// Public constructor for multiple files. - public LogfileReader (string[] fileNames, EncodingOptions encodingOptions, int bufferCount, int linesPerBuffer, MultiFileOptions multiFileOptions, ReaderType readerType, IPluginRegistry pluginRegistry, int maximumLineLength) - : this(fileNames, encodingOptions, true, bufferCount, linesPerBuffer, multiFileOptions, readerType, pluginRegistry, maximumLineLength) + public LogfileReader ( + string[] fileNames, + EncodingOptions encodingOptions, + int bufferCount, + int linesPerBuffer, + MultiFileOptions multiFileOptions, + ReaderType readerType, + IPluginRegistry pluginRegistry, + int maximumLineLength, + ILoadProgressReporter? progressReporter = null) + : this( + fileNames, + encodingOptions, + true, + bufferCount, + linesPerBuffer, + multiFileOptions, + readerType, + pluginRegistry, + maximumLineLength, + progressReporter) { // In this overload, we assume multiFile is always true. } // Single private constructor that contains the common initialization logic. - private LogfileReader (string[] fileNames, EncodingOptions encodingOptions, bool multiFile, int bufferCount, int linesPerBuffer, MultiFileOptions multiFileOptions, ReaderType readerType, IPluginRegistry pluginRegistry, int maximumLineLength) + private LogfileReader ( + string[] fileNames, + EncodingOptions encodingOptions, + bool multiFile, + int bufferCount, + int linesPerBuffer, + MultiFileOptions multiFileOptions, + ReaderType readerType, + IPluginRegistry pluginRegistry, + int maximumLineLength, + ILoadProgressReporter? progressReporter = null) { // Validate input: at least one file must be provided. if (fileNames == null || fileNames.Length < 1) @@ -104,7 +148,7 @@ private LogfileReader (string[] fileNames, EncodingOptions encodingOptions, bool _bufferPool = new LogBufferPool(_max_buffers * 2); - InitLruBuffers(); + BufferIndex = new BufferIndex(_max_buffers, _maxLinesPerBuffer); ILogFileInfo fileInfo = null; @@ -116,6 +160,19 @@ private LogfileReader (string[] fileNames, EncodingOptions encodingOptions, bool ? new RolloverFilenameHandler(GetLogFileInfo(_fileName), _multiFileOptions).GetNameList(_pluginRegistry) : [_fileName]; + if (progressReporter != null) + { + _progressReporter = progressReporter; + } + else + { + var reporter = new PeriodicProgressReporter(); + reporter.LoadFile += (_, e) => OnLoadFile(e); + reporter.LoadingStarted += (_, e) => OnLoadingStarted(e); + reporter.LoadingFinished += (_, _) => OnLoadingFinished(); + _progressReporter = reporter; + } + foreach (var name in names) { fileInfo = AddFile(name); @@ -159,6 +216,9 @@ private LogfileReader (string[] fileNames, EncodingOptions encodingOptions, bool #region Properties + /// For tests and diagnostics. + public BufferIndex BufferIndex { get; } + /// /// Gets the total number of lines contained in all buffers. /// @@ -170,39 +230,18 @@ public int LineCount { get { - if (_isLineCountDirty) - { - field = 0; - if (_bufferListLock.IsReadLockHeld || _bufferListLock.IsWriteLockHeld) - { - foreach (var buffer in _bufferList.Values) - { - field += buffer.LineCount; - } - } - else - { - AcquireBufferListReaderLock(); - try - { - foreach (var buffer in _bufferList.Values) - { - field += buffer.LineCount; - } - } - finally - { - ReleaseBufferListReaderLock(); - } - } + using var _ = BufferIndex.AcquireReadLock(); + return BufferIndex.TotalLineCount; + } - _isLineCountDirty = false; + private set + { + // Only used for resetting to 0. The actual count is computed by BufferIndex. + if (value == 0) + { + BufferIndex.MarkLineCountDirty(); } - - return field; } - - private set; } /// @@ -270,17 +309,15 @@ private EncodingOptions EncodingOptions //TODO: Make this private public void ReadFiles () { - _lastProgressUpdate = 0; FileSize = 0; - LineCount = 0; _isDeleted = false; - ClearLru(); - AcquireBufferListWriterLock(); - _bufferList.Clear(); - ReleaseBufferListWriterLock(); + try { + using var _ = BufferIndex.AcquireWriteLock(); + BufferIndex.ClearLru(_bufferPool); + foreach (var info in _logFileInfoList) { ReadToBufferList(info, 0, LineCount); @@ -330,21 +367,20 @@ public int ShiftBuffers () { _logger.Info(CultureInfo.InvariantCulture, "ShiftBuffers() begin for {0}{1}", _fileName, IsMultiFile ? " (MultiFile)" : ""); - AcquireBufferListWriterLock(); - - try + using var writeLock = BufferIndex.AcquireWriteLock(); { - ClearBufferState(); + BufferIndex.ResetThreadLocalCache(); var offset = 0; - _isLineCountDirty = true; + BufferIndex.MarkLineCountDirty(); lock (_monitor) { RolloverFilenameHandler rolloverHandler = new(_watchedILogFileInfo, _multiFileOptions); var fileNameList = rolloverHandler.GetNameList(_pluginRegistry); - ResetBufferCache(); + FileSize = 0; + LineCount = 0; IList lostILogFileInfoList = []; IList readNewILogFileInfoList = []; @@ -425,16 +461,16 @@ public int ShiftBuffers () } } - _logger.Info(CultureInfo.InvariantCulture, "Adjusting StartLine values in {0} buffers by offset {1}", _bufferList.Count, offset); - foreach (var buffer in _bufferList.Values.ToList()) + _logger.Info(CultureInfo.InvariantCulture, "Adjusting StartLine values in {0} buffers by offset {1}", BufferIndex.BufferCount, offset); + foreach (var buffer in BufferIndex.EnumerateBuffers()) { - SetNewStartLineForBuffer(buffer, buffer.StartLine - offset); + BufferIndex.UpdateStartLine(buffer, buffer.StartLine - offset); } #if DEBUG - if (_bufferList.Values.Count > 0) + if (BufferIndex.BufferCount > 0) { - _logger.Debug(CultureInfo.InvariantCulture, "First buffer now has StartLine {0}", _bufferList.Values[0].StartLine); + _logger.Debug(CultureInfo.InvariantCulture, "First buffer now has StartLine {0}", BufferIndex.GetBufferAt(0).StartLine); } #endif } @@ -471,68 +507,6 @@ public int ShiftBuffers () return offset; } - finally - { - ReleaseBufferListWriterLock(); - } - } - - /// - /// Acquires a read lock on the buffer list, waiting up to 10 seconds before forcing entry if the lock is not - /// immediately available. - /// - /// - /// If the read lock cannot be acquired within 10 seconds, the method will forcibly enter the lock and log a - /// warning. Callers should ensure that holding the read lock for extended periods does not block other operations. - /// - private void AcquireBufferListReaderLock () - { - if (!_bufferListLock.TryEnterReadLock(TimeSpan.FromSeconds(10))) - { - _logger.Warn("Reader lock wait timed out, forcing entry"); - _bufferListLock.EnterReadLock(); - } - } - - /// - /// Releases the reader lock on the buffer list, allowing other threads to acquire write access. - /// - /// - /// Call this method after completing operations that require read access to the buffer list. Failing to release the - /// reader lock may result in deadlocks or prevent other threads from obtaining write access. - /// - private void ReleaseBufferListReaderLock () - { - _bufferListLock.ExitReadLock(); - } - - /// - /// Releases the writer lock on the buffer list, allowing other threads to acquire the lock. - /// - /// - /// Call this method after completing operations that required exclusive access to the buffer list. Failing to - /// release the writer lock may result in deadlocks or reduced concurrency. - /// - private void ReleaseBufferListWriterLock () - { - _bufferListLock.ExitWriteLock(); - } - - /// - /// Acquires the writer lock for the buffer list, blocking the calling thread until the lock is obtained. - /// - /// - /// If the writer lock cannot be acquired within 10 seconds, a warning is logged and the method will continue to - /// wait until the lock becomes available. This method should be used to ensure exclusive access to the buffer list - /// when performing write operations. - /// - private void AcquireBufferListWriterLock () - { - if (!_bufferListLock.TryEnterWriteLock(TimeSpan.FromSeconds(10))) - { - _logger.Warn("Writer lock wait timed out"); - _bufferListLock.EnterWriteLock(); - } } //TODO Make Task Based @@ -565,15 +539,11 @@ public async Task GetLogLineMemoryWithWait (int lineNum) { // Fast path: if the buffer is in memory, skip the thread-pool hop entirely bool canFastPath = false; - AcquireBufferListReaderLock(); - try - { - var (logBuffer, _) = GetBufferForLineWithIndex(lineNum); - canFastPath = logBuffer is { IsDisposed: false }; - } - finally + + using (BufferIndex.AcquireReadLock()) { - ReleaseBufferListReaderLock(); + var logBufferEntry = BufferIndex.GetBufferForLineWithIndex(lineNum); + canFastPath = logBufferEntry.Buffer is { IsDisposed: false }; } if (canFastPath) @@ -618,8 +588,9 @@ public async Task GetLogLineMemoryWithWait (int lineNum) /// public string GetLogFileNameForLine (int lineNum) { - var logBuffer = GetBufferForLine(lineNum); - return logBuffer?.FileInfo.FullName; + using var _ = BufferIndex.AcquireReadLock(); + var logBufferEntry = BufferIndex.TryFindBuffer(lineNum); + return logBufferEntry.Buffer?.FileInfo.FullName; } /// @@ -629,8 +600,9 @@ public string GetLogFileNameForLine (int lineNum) /// public ILogFileInfo GetLogFileInfoForLine (int lineNum) { - var logBuffer = GetBufferForLine(lineNum); - return logBuffer?.FileInfo; + using var _ = BufferIndex.AcquireReadLock(); + var logBufferEntry = BufferIndex.TryFindBuffer(lineNum); + return logBufferEntry.Buffer?.FileInfo; } /// @@ -640,30 +612,9 @@ public ILogFileInfo GetLogFileInfoForLine (int lineNum) /// public int GetNextMultiFileLine (int lineNum) { - var result = -1; - AcquireBufferListReaderLock(); - - try - { - var (logBuffer, index) = GetBufferForLineWithIndex(lineNum); - if (logBuffer != null && index != -1) - { - for (var i = index; i < _bufferList.Values.Count; ++i) - { - if (_bufferList.Values[i].FileInfo != logBuffer.FileInfo) - { - result = _bufferList.Values[i].StartLine; - break; - } - } - } - } - finally - { - ReleaseBufferListReaderLock(); - } - - return result; + using var _ = BufferIndex.AcquireReadLock(); + var (found, startLine) = BufferIndex.TryGetNextFileStartLine(lineNum); + return found ? startLine : -1; } /// @@ -681,30 +632,9 @@ public int GetNextMultiFileLine (int lineNum) /// The starting line number of the previous file segment if one exists; otherwise, -1. public int GetPrevMultiFileLine (int lineNum) { - var result = -1; - AcquireBufferListReaderLock(); - - try - { - var (logBuffer, index) = GetBufferForLineWithIndex(lineNum); - if (logBuffer != null && index != -1) - { - for (var i = index; i >= 0; --i) - { - if (_bufferList.Values[i].FileInfo != logBuffer.FileInfo) - { - result = _bufferList.Values[i].StartLine + _bufferList.Values[i].LineCount; - break; - } - } - } - } - finally - { - ReleaseBufferListReaderLock(); - } - - return result; + using var _ = BufferIndex.AcquireReadLock(); + var (found, startLine) = BufferIndex.TryGetPrevFileStartLine(lineNum); + return found ? startLine : -1; } /// @@ -718,26 +648,15 @@ public int GetPrevMultiFileLine (int lineNum) /// public int GetRealLineNumForVirtualLineNum (int lineNum) { - var result = -1; - AcquireBufferListReaderLock(); - try - { - var (logBuffer, index) = GetBufferForLineWithIndex(lineNum); - if (logBuffer != null) - { - logBuffer = GetFirstBufferForFileByLogBuffer(logBuffer, index); - if (logBuffer != null) - { - result = lineNum - logBuffer.StartLine; - } - } - } - finally + using var _ = BufferIndex.AcquireReadLock(); + var logBufferEntry = BufferIndex.GetBufferForLineWithIndex(lineNum); + if (!logBufferEntry.Found) { - ReleaseBufferListReaderLock(); + return logBufferEntry.Index; } - return result; + var buffer = BufferIndex.GetFirstBufferForFile(logBufferEntry.Buffer, logBufferEntry.Index); + return buffer != null ? lineNum - buffer.StartLine : -1; } /// @@ -807,11 +726,11 @@ public void DeleteAllContent () } _logger.Info(CultureInfo.InvariantCulture, "Deleting all log buffers for {0}. Used mem: {1:N0}", Util.GetNameFromPath(_fileName), GC.GetTotalMemory(false)); - AcquireBufferListWriterLock(); - ClearBufferState(); - //AcquireDisposeWriterLock(); - foreach (var logBuffer in _bufferList.Values) + using var _ = BufferIndex.AcquireWriteLock(); + BufferIndex.ResetThreadLocalCache(); + + foreach (var logBuffer in BufferIndex.EnumerateBuffers()) { if (!logBuffer.IsDisposed) { @@ -819,23 +738,11 @@ public void DeleteAllContent () } } - _lruCacheDict.Clear(); - _bufferList.Clear(); - - //ReleaseDisposeWriterLock(); - ReleaseBufferListWriterLock(); + BufferIndex.Clear(); _contentDeleted = true; _logger.Info(CultureInfo.InvariantCulture, "Deleting complete. Used mem: {0:N0}", GC.GetTotalMemory(false)); } - /// - /// Clears the Buffer so that no stale buffer references are kept - /// - private void ClearBufferState () - { - _lastBufferIndex.Value = -1; - } - /// /// Explicit change the encoding. /// @@ -844,8 +751,9 @@ public void ChangeEncoding (Encoding encoding) { CurrentEncoding = encoding; EncodingOptions.Encoding = encoding; - ResetBufferCache(); - ClearLru(); + FileSize = 0; + using var _ = BufferIndex.AcquireWriteLock(); + BufferIndex.ClearLru(_bufferPool); } /// @@ -857,15 +765,6 @@ public IList GetLogFileInfoList () return _logFileInfoList; } - /// - /// For unit tests only - /// - /// - public IList GetBufferList () - { - return _bufferList.Values; - } - #endregion #region Internals @@ -882,12 +781,10 @@ public IList GetBufferList () /// The zero-based line number for which buffer information is logged. public void LogBufferInfoForLine (int lineNum) { - AcquireBufferListReaderLock(); - - try + using var readLock = BufferIndex.AcquireReadLock(); { - var (buffer, _) = GetBufferForLineWithIndex(lineNum); - if (buffer == null) + var logBufferEntry = BufferIndex.GetBufferForLineWithIndex(lineNum); + if (!logBufferEntry.Found) { _logger.Error("Cannot find buffer for line {0}, file: {1}{2}", lineNum, _fileName, IsMultiFile ? " (MultiFile)" : ""); return; @@ -895,14 +792,10 @@ public void LogBufferInfoForLine (int lineNum) _logger.Info(CultureInfo.InvariantCulture, "-----------------------------------"); _logger.Info(CultureInfo.InvariantCulture, "Buffer info for line {0}", lineNum); - DumpBufferInfos(buffer); - _logger.Info(CultureInfo.InvariantCulture, "File pos for current line: {0}", buffer.GetFilePosForLineOfBlock(lineNum - buffer.StartLine)); + DumpBufferInfos(logBufferEntry.Buffer); + _logger.Info(CultureInfo.InvariantCulture, "File pos for current line: {0}", logBufferEntry.Buffer.GetFilePosForLineOfBlock(lineNum - logBufferEntry.Buffer.StartLine)); _logger.Info(CultureInfo.InvariantCulture, "-----------------------------------"); } - finally - { - ReleaseBufferListReaderLock(); - } } /// @@ -916,34 +809,35 @@ public void LogBufferInfoForLine (int lineNum) public void LogBufferDiagnostic () { _logger.Info(CultureInfo.InvariantCulture, "-------- Buffer diagnostics -------"); - var cacheCount = _lruCacheDict.Count; + var cacheCount = BufferIndex.LruCacheCount; _logger.Info(CultureInfo.InvariantCulture, "LRU entries: {0}", cacheCount); - AcquireBufferListReaderLock(); - _logger.Info(CultureInfo.InvariantCulture, "File: {0}\r\nBuffer count: {1}\r\nDisposed buffers: {2}", _fileName, _bufferList.Count, _bufferList.Count - cacheCount); - var lineNum = 0; - long disposeSum = 0; - long maxDispose = 0; - long minDispose = int.MaxValue; - - for (var i = 0; i < _bufferList.Values.Count; ++i) + using var readLock = BufferIndex.AcquireReadLock(); { - var buffer = _bufferList.Values[i]; - if (buffer.StartLine != lineNum) + _logger.Info(CultureInfo.InvariantCulture, "File: {0}\r\nBuffer count: {1}\r\nDisposed buffers: {2}", _fileName, BufferIndex.BufferCount, BufferIndex.BufferCount - cacheCount); + var lineNum = 0; + long disposeSum = 0; + long maxDispose = 0; + long minDispose = int.MaxValue; + + for (var i = 0; i < BufferIndex.BufferCount; ++i) { - _logger.Error("Start line of buffer is: {0}, expected: {1}", buffer.StartLine, lineNum); - _logger.Info(CultureInfo.InvariantCulture, "Info of buffer follows:"); - DumpBufferInfos(buffer); + var buffer = BufferIndex.GetBufferAt(i); + if (buffer.StartLine != lineNum) + { + _logger.Error("Start line of buffer is: {0}, expected: {1}", buffer.StartLine, lineNum); + _logger.Info(CultureInfo.InvariantCulture, "Info of buffer follows:"); + DumpBufferInfos(buffer); + } + + lineNum += buffer.LineCount; + disposeSum += buffer.DisposeCount; + maxDispose = Math.Max(maxDispose, buffer.DisposeCount); + minDispose = Math.Min(minDispose, buffer.DisposeCount); } - lineNum += buffer.LineCount; - disposeSum += buffer.DisposeCount; - maxDispose = Math.Max(maxDispose, buffer.DisposeCount); - minDispose = Math.Min(minDispose, buffer.DisposeCount); + _logger.Info(CultureInfo.InvariantCulture, "Dispose count sum is: {0}\r\nMin dispose count is: {1}\r\nMax dispose count is: {2}\r\n-----------------------------------", disposeSum, minDispose, maxDispose); } - - ReleaseBufferListReaderLock(); - _logger.Info(CultureInfo.InvariantCulture, "Dispose count sum is: {0}\r\nMin dispose count is: {1}\r\nMax dispose count is: {2}\r\n-----------------------------------", disposeSum, minDispose, maxDispose); } #endif @@ -985,14 +879,13 @@ public ILogLineMemory[] GetLogLineMemories (int startLine, int count) var result = new ILogLineMemory[count]; var filled = 0; - AcquireBufferListReaderLock(); - try + using var readLock = BufferIndex.AcquireReadLock(); { var lineNum = startLine; while (filled < count) { - var (logBuffer, _) = GetBufferForLineWithIndex(lineNum); - if (logBuffer == null) + var logBufferEntry = BufferIndex.GetBufferForLineWithIndex(lineNum); + if (!logBufferEntry.Found) { break; } @@ -1001,24 +894,24 @@ public ILogLineMemory[] GetLogLineMemories (int startLine, int count) var lockTaken = false; try { - logBuffer.AcquireContentLock(ref lockTaken); + logBufferEntry.Buffer.AcquireContentLock(ref lockTaken); - if (logBuffer.IsDisposed) + if (logBufferEntry.Buffer.IsDisposed) { - lock (logBuffer.FileInfo) + lock (logBufferEntry.Buffer.FileInfo) { - ReReadBuffer(logBuffer); + ReReadBuffer(logBufferEntry.Buffer); } } // Copy lines from this buffer - var bufferOffset = lineNum - logBuffer.StartLine; - var availableInBuffer = logBuffer.LineCount - bufferOffset; + var bufferOffset = lineNum - logBufferEntry.Buffer.StartLine; + var availableInBuffer = logBufferEntry.Buffer.LineCount - bufferOffset; var toCopy = Math.Min(count - filled, availableInBuffer); for (var i = 0; i < toCopy; i++) { - result[filled + i] = logBuffer.GetLineMemoryOfBlock(bufferOffset + i); + result[filled + i] = logBufferEntry.Buffer.GetLineMemoryOfBlock(bufferOffset + i); } filled += toCopy; @@ -1028,15 +921,11 @@ public ILogLineMemory[] GetLogLineMemories (int startLine, int count) { if (lockTaken) { - logBuffer.ReleaseContentLock(); + logBufferEntry.Buffer.ReleaseContentLock(); } } } } - finally - { - ReleaseBufferListReaderLock(); - } // Trim if we got fewer lines than requested if (filled < count) @@ -1071,11 +960,10 @@ private ValueTask GetLogLineMemoryInternal (int lineNum) return new ValueTask(line); } - AcquireBufferListReaderLock(); - try + using var readLock = BufferIndex.AcquireReadLock(); { - var (logBuffer, _) = GetBufferForLineWithIndex(lineNum); - if (logBuffer == null) + var logBufferEntry = BufferIndex.GetBufferForLineWithIndex(lineNum); + if (!logBufferEntry.Found) { _logger.Error("Cannot find buffer for line {0}, file: {1}{2}", lineNum, _fileName, IsMultiFile ? " (MultiFile)" : ""); return default; @@ -1084,17 +972,17 @@ private ValueTask GetLogLineMemoryInternal (int lineNum) var lockTaken = false; try { - logBuffer.AcquireContentLock(ref lockTaken); + logBufferEntry.Buffer.AcquireContentLock(ref lockTaken); - if (logBuffer.IsDisposed) + if (logBufferEntry.Buffer.IsDisposed) { - lock (logBuffer.FileInfo) + lock (logBufferEntry.Buffer.FileInfo) { - ReReadBuffer(logBuffer); + ReReadBuffer(logBufferEntry.Buffer); } } - var line = logBuffer.GetLineMemoryOfBlock(lineNum - logBuffer.StartLine); + var line = logBufferEntry.Buffer.GetLineMemoryOfBlock(lineNum - logBufferEntry.Buffer.StartLine); return line.HasValue ? new ValueTask(line.Value) : default; @@ -1103,28 +991,10 @@ private ValueTask GetLogLineMemoryInternal (int lineNum) { if (lockTaken) { - logBuffer.ReleaseContentLock(); + logBufferEntry.Buffer.ReleaseContentLock(); } } } - finally - { - ReleaseBufferListReaderLock(); - } - } - - /// - /// Initializes the internal data structures used for least recently used (LRU) buffer management. - /// - /// - /// Call this method to reset or prepare the LRU buffer cache before use. This method clears any existing buffer - /// state and sets up the cache to track buffer usage according to the configured maximum buffer count. - /// - private void InitLruBuffers () - { - ClearBufferState(); - _bufferList = []; - _lruCacheDict = new ConcurrentDictionary(concurrencyLevel: Environment.ProcessorCount, capacity: _max_buffers + 1); } /// @@ -1140,19 +1010,6 @@ private void StartGCThread () _garbageCollectorTask = Task.Run(GarbageCollectorThreadProc, _cts.Token); } - /// - /// Resets the internal buffer cache, clearing any stored file size and line count information. - /// - /// - /// Call this method to reinitialize the buffer cache state, typically before reloading or reprocessing file data. - /// After calling this method, any previously cached file size or line count values will be lost. - /// - private void ResetBufferCache () - { - FileSize = 0; - LineCount = 0; - } - /// /// Releases resources associated with open log files and resets related state information. /// @@ -1195,7 +1052,7 @@ private ILogFileInfo GetLogFileInfo (string fileNameOrUri) //TODO: I changed to private void ReplaceBufferInfos (ILogFileInfo oldLogFileInfo, ILogFileInfo newLogFileInfo) { _logger.Debug(CultureInfo.InvariantCulture, "ReplaceBufferInfos() " + oldLogFileInfo.FullName + " -> " + newLogFileInfo.FullName); - foreach (var buffer in _bufferList.Values) + foreach (var buffer in BufferIndex.EnumerateBuffers()) { if (buffer.FileInfo == oldLogFileInfo) { @@ -1223,13 +1080,13 @@ private void ReplaceBufferInfos (ILogFileInfo oldLogFileInfo, ILogFileInfo newLo private (int StartLine, int LineCount)? DeleteBuffersForInfo (ILogFileInfo iLogFileInfo, bool matchNamesOnly) { _logger.Info($"Deleting buffers for file {iLogFileInfo.FullName}"); - ClearBufferState(); + BufferIndex.ResetThreadLocalCache(); (int StartLine, int LineCount)? lastRemovedInfo = null; IList deleteList = []; if (matchNamesOnly) { - foreach (var buffer in _bufferList.Values) + foreach (var buffer in BufferIndex.EnumerateBuffers()) { if (buffer.FileInfo.FullName.Equals(iLogFileInfo.FullName, StringComparison.Ordinal)) { @@ -1240,7 +1097,7 @@ private void ReplaceBufferInfos (ILogFileInfo oldLogFileInfo, ILogFileInfo newLo } else { - foreach (var buffer in _bufferList.Values) + foreach (var buffer in BufferIndex.EnumerateBuffers()) { if (buffer.FileInfo == iLogFileInfo) { @@ -1252,7 +1109,7 @@ private void ReplaceBufferInfos (ILogFileInfo oldLogFileInfo, ILogFileInfo newLo foreach (var buffer in deleteList) { - RemoveFromBufferList(buffer); + _ = BufferIndex.Remove(buffer); var lockTaken = false; try @@ -1281,22 +1138,6 @@ private void ReplaceBufferInfos (ILogFileInfo oldLogFileInfo, ILogFileInfo newLo return lastRemovedInfo; } - /// - /// Removes the specified log buffer from the internal buffer list and associated cache. The caller must have - /// _writer locks for lruCache and buffer list! - /// - /// - /// This method must be called only when the appropriate write locks for both the LRU cache and buffer list are - /// held. Removing a buffer that is not present has no effect. - /// - /// The log buffer to remove from the buffer list and cache. Must not be null. - private void RemoveFromBufferList (LogBuffer buffer) - { - Util.AssertTrue(_bufferListLock.IsWriteLockHeld, "No _writer lock for buffer list"); - _ = _lruCacheDict.TryRemove(buffer.StartLine, out _); - _ = _bufferList.Remove(buffer.StartLine); - } - /// /// Reads log lines from the specified log file starting at the given file position and line number, and populates /// the internal buffer list with the read data. @@ -1326,30 +1167,22 @@ private void ReadToBufferList (ILogFileInfo logFileInfo, long filePos, int start var lineNum = startLine; LogBuffer logBuffer; - AcquireBufferListUpgradeableReadLock(); - - try + using var upgradeabelLock = BufferIndex.AcquireUpgradeableReadLock(); { - if (_bufferList.Count == 0) + if (BufferIndex.BufferCount == 0) { logBuffer = _bufferPool.Rent(logFileInfo, _maxLinesPerBuffer); logBuffer.StartLine = startLine; logBuffer.StartPos = filePos; - UpgradeBufferlistLockToWriterLock(); - - try - { - AddBufferToList(logBuffer); - } - finally + using (upgradeabelLock.UpgradeToWrite()) { - DowngradeBufferListLockFromWriterLock(); + BufferIndex.Add(logBuffer); } } else { - logBuffer = _bufferList.Values[_bufferList.Count - 1]; + logBuffer = BufferIndex.GetLastBuffer(); if (!logBuffer.FileInfo.FullName.Equals(logFileInfo.FullName, StringComparison.Ordinal)) { @@ -1357,15 +1190,9 @@ private void ReadToBufferList (ILogFileInfo logFileInfo, long filePos, int start logBuffer.StartLine = startLine; logBuffer.StartPos = filePos; - UpgradeBufferlistLockToWriterLock(); - - try - { - AddBufferToList(logBuffer); - } - finally + using (upgradeabelLock.UpgradeToWrite()) { - DowngradeBufferListLockFromWriterLock(); + BufferIndex.Add(logBuffer); } } @@ -1388,10 +1215,6 @@ private void ReadToBufferList (ILogFileInfo logFileInfo, long filePos, int start } } } - finally - { - ReleaseBufferListUpgradeableReadLock(); - } Monitor.Enter(logBuffer); try @@ -1419,19 +1242,30 @@ private void ReadToBufferList (ILogFileInfo logFileInfo, long filePos, int start lineCount++; + // Add the line to the CURRENT buffer BEFORE any buffer rotation. + // The lineMemory slice is backed by the allocator's current char[] block, + // so it must be added to the buffer that will own those blocks after DetachBlocks(). + var logLine = new LogLine(lineMemory, logBuffer.StartLine + logBuffer.LineCount); + logBuffer.AddLine(logLine, filePos); + filePos = reader.Position; + lineNum++; + if (lineCount > _maxLinesPerBuffer && reader.IsBufferComplete) { - //Rate Limited Progrress - var now = Environment.TickCount64; - bool shouldFireLoadFileEvent = (now - _lastProgressUpdate) >= PROGRESS_UPDATE_INTERVAL_MS; + _progressReporter.ReportProgress(logFileInfo.FullName, filePos, logFileInfo.Length); + + logBuffer.Size = filePos - logBuffer.StartPos; - if (shouldFireLoadFileEvent) + // Detach char blocks from the reader's allocator and attach to the completed buffer. + // Must happen before Monitor.Exit so the buffer is still exclusively owned. + if (reader is PositionAwareStreamReaderSystem systemDetachBlockReader) { - OnLoadFile(new LoadFileEventArgs(logFileInfo.FullName, filePos, false, logFileInfo.Length, false)); - _lastProgressUpdate = now; + logBuffer.AttachCharBlocks(systemDetachBlockReader.BlockAllocator.DetachBlocks()); + } + else if (reader is PositionAwareStreamReaderDirect directReader) + { + logBuffer.AttachCharBlocks(directReader.DetachBlocks()); } - - logBuffer.Size = filePos - logBuffer.StartPos; Monitor.Exit(logBuffer); try @@ -1441,15 +1275,9 @@ private void ReadToBufferList (ILogFileInfo logFileInfo, long filePos, int start newBuffer.StartPos = filePos; newBuffer.PrevBuffersDroppedLinesSum = droppedLines; - AcquireBufferListWriterLock(); - - try - { - AddBufferToList(newBuffer); - } - finally + using (upgradeabelLock.UpgradeToWrite()) { - ReleaseBufferListWriterLock(); + BufferIndex.Add(newBuffer); } logBuffer = newBuffer; @@ -1463,22 +1291,27 @@ private void ReadToBufferList (ILogFileInfo logFileInfo, long filePos, int start } } - var logLine = new LogLine(lineMemory, logBuffer.StartLine + logBuffer.LineCount); - logBuffer.AddLine(logLine, filePos); - filePos = reader.Position; - lineNum++; - (success, lineMemory, wasDropped) = ReadLineMemory(reader, logBuffer.StartLine + logBuffer.LineCount, logBuffer.StartLine + logBuffer.LineCount + droppedLines); } logBuffer.Size = filePos - logBuffer.StartPos; + + // Attach remaining blocks to the final buffer + if (reader is PositionAwareStreamReaderSystem systemDetachBlockReader2) + { + logBuffer.AttachCharBlocks(systemDetachBlockReader2.BlockAllocator.DetachBlocks()); + } + else if (reader is PositionAwareStreamReaderDirect directReader) + { + logBuffer.AttachCharBlocks(directReader.DetachBlocks()); + } } finally { Monitor.Exit(logBuffer); } - _isLineCountDirty = true; + BufferIndex.MarkLineCountDirty(); FileSize = reader.Position; // Reader may have detected another encoding @@ -1486,7 +1319,7 @@ private void ReadToBufferList (ILogFileInfo logFileInfo, long filePos, int start if (!_shouldStop) { - OnLoadFile(new LoadFileEventArgs(logFileInfo.FullName, filePos, true, _fileLength, false)); + _progressReporter.ReportComplete(logFileInfo.FullName, filePos, _fileLength); } } catch (IOException ioex) @@ -1499,122 +1332,6 @@ private void ReadToBufferList (ILogFileInfo logFileInfo, long filePos, int start } } - /// - /// Adds the specified log buffer to the internal buffer list and updates its position in the least recently used - /// (LRU) cache. - /// - /// The log buffer to add to the buffer list. Cannot be null. - private void AddBufferToList (LogBuffer logBuffer) - { -#if DEBUG - _logger.Debug(CultureInfo.InvariantCulture, "AddBufferToList(): {0}/{1}/{2}", logBuffer.StartLine, logBuffer.LineCount, logBuffer.FileInfo.FullName); -#endif - _bufferList[logBuffer.StartLine] = logBuffer; - UpdateLruCache(logBuffer); - } - - /// - /// Updates the least recently used (LRU) cache with the specified log buffer, adding it if it does not already - /// exist or marking it as recently used if it does. - /// - /// - /// If the specified log buffer is not already present in the cache, it is added. If it is present, its usage is - /// updated to reflect recent access. This method is thread-safe and manages cache locks internally. - /// - /// The log buffer to add to or update in the LRU cache. Cannot be null. - private void UpdateLruCache (LogBuffer logBuffer) - { - var cacheEntry = _lruCacheDict.GetOrAdd( - logBuffer.StartLine, - static (_, buf) => new LogBufferCacheEntry { LogBuffer = buf }, - logBuffer); - - cacheEntry.Touch(); - } - - /// - /// Sets a new start line in the given buffer and updates the LRU cache, if the buffer is present in the cache. The - /// caller must have write lock for 'lruCacheDictLock'; - /// - /// - /// - private void SetNewStartLineForBuffer (LogBuffer logBuffer, int newLineNum) - { - var hadCache = _lruCacheDict.TryRemove(logBuffer.StartLine, out var cacheEntry); - - _ = _bufferList.Remove(logBuffer.StartLine); - logBuffer.StartLine = newLineNum; - _bufferList[newLineNum] = logBuffer; - - if (hadCache) - { - _ = _lruCacheDict.TryAdd(logBuffer.StartLine, cacheEntry); - } - } - - /// - /// Removes least recently used entries from the LRU cache to maintain the cache size within the configured limit. - /// - /// - /// This method is intended to be called when the LRU cache exceeds its maximum allowed size. It removes the least - /// recently used entries to free up resources and ensure optimal cache performance. The method is not thread-safe - /// and should be called only when appropriate locks are held to prevent concurrent modifications. - /// - private void GarbageCollectLruCache () - { -#if DEBUG - long startTime = Environment.TickCount; -#endif - _logger.Debug(CultureInfo.InvariantCulture, "Starting garbage collection"); - var threshold = 10; - var diff = 0; - if (_lruCacheDict.Count - (_max_buffers + threshold) > 0) - { - diff = _lruCacheDict.Count - _max_buffers; -#if DEBUG - if (diff > 0) - { - _logger.Info(CultureInfo.InvariantCulture, "Removing {0} entries from LRU cache for {1}", diff, Util.GetNameFromPath(_fileName)); - } -#endif - // Snapshot values and sort by timestamp (ascending = least recently used first) - var entries = _lruCacheDict.ToArray(); - Array.Sort(entries, static (a, b) => a.Value.LastUseTimeStamp.CompareTo(b.Value.LastUseTimeStamp)); - - for (var i = 0; i < diff && i < entries.Length; ++i) - { - var kvp = entries[i]; - if (_lruCacheDict.TryRemove(kvp.Key, out var removed)) - { - var lockTaken = false; - try - { - removed.LogBuffer.AcquireContentLock(ref lockTaken); - // Evict content but preserve metadata (LineCount, StartLine, etc.) - // so the buffer remains findable in _bufferList lookups. - // Do NOT return to pool — the buffer is still referenced by _bufferList. - removed.LogBuffer.EvictContent(); - } - finally - { - if (lockTaken) - { - removed.LogBuffer.ReleaseContentLock(); - } - } - } - } - } - -#if DEBUG - if (diff > 0) - { - long endTime = Environment.TickCount; - _logger.Info(CultureInfo.InvariantCulture, "Garbage collector time: " + (endTime - startTime) + " ms."); - } -#endif - } - /// /// Executes the background thread procedure responsible for periodically triggering garbage collection of the least /// recently used (LRU) cache while the thread is active. @@ -1637,42 +1354,10 @@ private async Task GarbageCollectorThreadProc () break; } - GarbageCollectLruCache(); + BufferIndex.EvictLeastRecentlyUsed(); } } - /// - /// Clears all entries from the least recently used (LRU) cache and releases associated resources. - /// - /// - /// Call this method to remove all items from the LRU cache and dispose of their contents. This operation is - /// typically used to free memory or reset the cache state. The method is not thread-safe and should be called only - /// when appropriate synchronization is ensured. - /// - private void ClearLru () - { - _logger.Info(CultureInfo.InvariantCulture, "Clearing LRU cache."); - foreach (var entry in _lruCacheDict.Values) - { - var lockTaken = false; - try - { - entry.LogBuffer.AcquireContentLock(ref lockTaken); - _bufferPool.Return(entry.LogBuffer); - } - finally - { - if (lockTaken) - { - entry.LogBuffer.ReleaseContentLock(); - } - } - } - - _lruCacheDict.Clear(); - _logger.Info(CultureInfo.InvariantCulture, "Clearing done."); - } - /// /// Re-reads the contents of the specified log buffer from its associated file, updating its lines and dropped line /// count as necessary. @@ -1691,10 +1376,10 @@ private void ReReadBuffer (LogBuffer logBuffer) #endif lock (_logBufferLock) { - Stream fileStream = null; + Stream openendFileStream; try { - fileStream = logBuffer.FileInfo.OpenStream(); + openendFileStream = logBuffer.FileInfo.OpenStream(); } catch (IOException e) { @@ -1702,193 +1387,72 @@ private void ReReadBuffer (LogBuffer logBuffer) return; } - try + using Stream fileStream = openendFileStream; { //TODO LogStream Reader has to be changed to ILogStreamReaderMemory var reader = GetLogStreamReader(fileStream, EncodingOptions) as ILogStreamReaderMemory; - var filePos = logBuffer.StartPos; - reader.Position = logBuffer.StartPos; - var maxLinesCount = logBuffer.LineCount; - var lineCount = 0; - var dropCount = logBuffer.PrevBuffersDroppedLinesSum; - logBuffer.ClearLines(); - - var (success, lineMemory, wasDropped) = ReadLineMemory(reader, logBuffer.StartLine + logBuffer.LineCount, logBuffer.StartLine + logBuffer.LineCount + dropCount); - - while (success) + using var readerDisposabel = reader as IDisposable; { - if (lineCount >= maxLinesCount) - { - break; - } + var filePos = logBuffer.StartPos; + reader.Position = logBuffer.StartPos; + var maxLinesCount = logBuffer.LineCount; + var lineCount = 0; + var dropCount = logBuffer.PrevBuffersDroppedLinesSum; + logBuffer.ClearLines(); - if (wasDropped) - { - dropCount++; - (success, lineMemory, wasDropped) = ReadLineMemory(reader, logBuffer.StartLine + logBuffer.LineCount, logBuffer.StartLine + logBuffer.LineCount + dropCount); - continue; - } - - LogLine logLine = new(lineMemory, logBuffer.StartLine + logBuffer.LineCount); + var (success, lineMemory, wasDropped) = ReadLineMemory(reader, logBuffer.StartLine + logBuffer.LineCount, logBuffer.StartLine + logBuffer.LineCount + dropCount); - logBuffer.AddLine(logLine, filePos); - filePos = reader.Position; - lineCount++; - - (success, lineMemory, wasDropped) = ReadLineMemory(reader, logBuffer.StartLine + logBuffer.LineCount, logBuffer.StartLine + logBuffer.LineCount + dropCount); - } - - if (maxLinesCount != logBuffer.LineCount) - { - _logger.Warn(CultureInfo.InvariantCulture, "LineCount in buffer differs after re-reading. old={0}, new={1}", maxLinesCount, logBuffer.LineCount); - } + while (success) + { + if (lineCount >= maxLinesCount) + { + break; + } - if (dropCount - logBuffer.PrevBuffersDroppedLinesSum != logBuffer.DroppedLinesCount) - { - _logger.Warn(CultureInfo.InvariantCulture, "DroppedLinesCount in buffer differs after re-reading. old={0}, new={1}", logBuffer.DroppedLinesCount, dropCount); - logBuffer.DroppedLinesCount = dropCount - logBuffer.PrevBuffersDroppedLinesSum; - } + if (wasDropped) + { + dropCount++; + (success, lineMemory, wasDropped) = ReadLineMemory(reader, logBuffer.StartLine + logBuffer.LineCount, logBuffer.StartLine + logBuffer.LineCount + dropCount); + continue; + } - GC.KeepAlive(fileStream); - } - catch (IOException e) - { - _logger.Warn(e); - } - finally - { - fileStream.Close(); - } - } - } + LogLine logLine = new(lineMemory, logBuffer.StartLine + logBuffer.LineCount); - /// - /// Core buffer lookup without acquiring _bufferListLock. The caller MUST already hold a read or write lock - /// on _bufferListLock. - /// - private (LogBuffer? Buffer, int Index) GetBufferForLineWithIndex (int lineNum) - { -#if DEBUG - Util.AssertTrue( - _bufferListLock.IsReadLockHeld || _bufferListLock.IsUpgradeableReadLockHeld || _bufferListLock.IsWriteLockHeld, - "No lock held for buffer list in GetBufferForLineWithIndex"); - long startTime = Environment.TickCount; -#endif - var arr = _bufferList.Values; - var count = arr.Count; + logBuffer.AddLine(logLine, filePos); + filePos = reader.Position; + lineCount++; - if (count == 0) - { - return (null, -1); - } + (success, lineMemory, wasDropped) = ReadLineMemory(reader, logBuffer.StartLine + logBuffer.LineCount, logBuffer.StartLine + logBuffer.LineCount + dropCount); + } - // Layer 0: Last buffer cache — O(1) for sequential access - var lastIdx = _lastBufferIndex.Value; - if (lastIdx >= 0 && lastIdx < count) - { - var buf = arr[lastIdx]; - if ((uint)(lineNum - buf.StartLine) < (uint)buf.LineCount) - { - //dont UpdateLRUCache, the cache has not changed in layer 0 - return (buf, lastIdx); - } + // Attach char blocks from the reader to the re-read buffer + if (reader is PositionAwareStreamReaderSystem systemReader) + { + logBuffer.AttachCharBlocks(systemReader.BlockAllocator.DetachBlocks()); + } + else if (reader is PositionAwareStreamReaderDirect directReader) + { + logBuffer.AttachCharBlocks(directReader.DetachBlocks()); + } - // Layer 1: Adjacent buffer prediction — O(1) for buffer boundary crossings - if (lastIdx + 1 < count) - { - var next = arr[lastIdx + 1]; - if ((uint)(lineNum - next.StartLine) < (uint)next.LineCount) - { - _lastBufferIndex.Value = lastIdx + 1; - UpdateLruCache(next); - return (next, lastIdx + 1); - } - } + if (maxLinesCount != logBuffer.LineCount) + { + _logger.Warn(CultureInfo.InvariantCulture, "LineCount in buffer differs after re-reading. old={0}, new={1}", maxLinesCount, logBuffer.LineCount); + } - if (lastIdx - 1 >= 0) - { - var prev = arr[lastIdx - 1]; - if ((uint)(lineNum - prev.StartLine) < (uint)prev.LineCount) - { - _lastBufferIndex.Value = lastIdx - 1; - UpdateLruCache(prev); - return (prev, lastIdx - 1); + if (dropCount - logBuffer.PrevBuffersDroppedLinesSum != logBuffer.DroppedLinesCount) + { + _logger.Warn(CultureInfo.InvariantCulture, "DroppedLinesCount in buffer differs after re-reading. old={0}, new={1}", logBuffer.DroppedLinesCount, dropCount); + logBuffer.DroppedLinesCount = dropCount - logBuffer.PrevBuffersDroppedLinesSum; + } } - } - } - - // Layer 2: Direct mapping guess — O(1) speculative for uniform buffers - var guess = lineNum / _maxLinesPerBuffer; - if ((uint)guess < (uint)count) - { - var buf = arr[guess]; - if ((uint)(lineNum - buf.StartLine) < (uint)buf.LineCount) - { - _lastBufferIndex.Value = guess; - UpdateLruCache(buf); - return (buf, guess); - } - } - // Layer 3: Branchless binary search with power-of-two strides - var step = HighestPowerOfTwo(count); - var idx = (arr[step - 1].StartLine <= lineNum) ? count - step : 0; - - for (step >>= 1; step > 0; step >>= 1) - { - var probe = idx + step; - if (probe < count && arr[probe - 1].StartLine <= lineNum) - { - idx = probe; - } - } - - // idx is now the buffer index — verify bounds - if (idx < count) - { - var buf = arr[idx]; - if ((uint)(lineNum - buf.StartLine) < (uint)buf.LineCount) - { - _lastBufferIndex.Value = idx; - UpdateLruCache(buf); - return (buf, idx); + GC.KeepAlive(fileStream); } } -#if DEBUG - long endTime = Environment.TickCount; - _logger.Debug($"GetBufferForLineWithIndex({lineNum}) duration: {endTime - startTime} ms."); -#endif - return (null, -1); } - /// - /// Retrieves the log buffer that contains the specified line number. - /// - /// - /// The zero-based line number for which to retrieve the corresponding log buffer. Must be greater than or equal to - /// zero. - /// - /// - /// The instance that contains the specified line number, or if no - /// such buffer exists. - /// - private LogBuffer GetBufferForLine (int lineNum) - { - AcquireBufferListReaderLock(); - try - { - var (buffer, _) = GetBufferForLineWithIndex(lineNum); - return buffer; - } - finally - { - ReleaseBufferListReaderLock(); - } - } - - private static int HighestPowerOfTwo (int n) => 1 << (31 - int.LeadingZeroCount(n)); - private void GetLineMemoryFinishedCallback (ILogLineMemory line) { _isFailModeCheckCallPending = false; @@ -1901,42 +1465,6 @@ private void GetLineMemoryFinishedCallback (ILogLineMemory line) _logger.Debug(CultureInfo.InvariantCulture, "'isLogLineCallPending' flag was reset."); } - /// - /// Finds the first buffer in the buffer list that is associated with the same file as the specified log buffer, - /// searching backwards from the given buffer. - /// - /// - /// This method searches backwards from the specified buffer in the buffer list to locate the earliest buffer - /// associated with the same file. The search is inclusive of the starting buffer. - /// - /// The log buffer from which to begin the search. Must not be null. - /// - /// The first LogBuffer in the buffer list that is associated with the same file as the specified buffer, searching - /// in reverse order from the given buffer. Returns null if the specified buffer is not found in the buffer list. - /// - private LogBuffer GetFirstBufferForFileByLogBuffer (LogBuffer logBuffer, int index) - { - var info = logBuffer.FileInfo; - if (index == -1) - { - return null; - } - - var resultBuffer = logBuffer; - while (true) - { - index--; - if (index < 0 || _bufferList.Values[index].FileInfo != info) - { - break; - } - - resultBuffer = _bufferList.Values[index]; - } - - return resultBuffer; - } - /// /// Monitors the specified log file for changes and processes updates in a background thread. /// @@ -1954,11 +1482,11 @@ private async Task MonitorThreadProc () try { - OnLoadingStarted(new LoadFileEventArgs(_fileName, 0, false, 0, false)); + _progressReporter.ReportLoadingStarted(_fileName); ReadFiles(); if (!_isDeleted) { - OnLoadingFinished(); + _progressReporter.ReportLoadingFinished(); } } catch (Exception e) @@ -2103,7 +1631,7 @@ private void FireChangeEvent () else { // Trigger "new file" handling (reload) - OnLoadFile(new LoadFileEventArgs(_fileName, 0, true, _fileLength, true)); + _progressReporter.ReportNewFile(_fileName, 0, _fileLength); if (_isDeleted) { @@ -2180,6 +1708,7 @@ private ILogStreamReader CreateLogStreamReader (Stream stream, EncodingOptions e { ReaderType.Legacy => new PositionAwareStreamReaderLegacy(stream, encodingOptions, _maximumLineLength), ReaderType.System => new PositionAwareStreamReaderSystem(stream, encodingOptions, _maximumLineLength), + ReaderType.SystemDirect => new PositionAwareStreamReaderDirect(stream, encodingOptions, _maximumLineLength), //Default will be System _ => new PositionAwareStreamReaderSystem(stream, encodingOptions, _maximumLineLength), }; @@ -2292,67 +1821,6 @@ private bool ReadLine (ILogStreamReader reader, int lineNum, int realLineNum, ou return (true, lineMemory, false); } - /// - /// Acquires an upgradeable read lock on the buffer list, waiting up to 10 seconds before blocking indefinitely if - /// the lock is not immediately available. - /// - /// - /// This method ensures that the calling thread holds an upgradeable read lock on the buffer list. If the lock - /// cannot be acquired within 10 seconds, a warning is logged and the method blocks until the lock becomes - /// available. Use this method when a read lock is needed with the potential to upgrade to a write lock. - /// - private void AcquireBufferListUpgradeableReadLock () - { - if (!_bufferListLock.TryEnterUpgradeableReadLock(TimeSpan.FromSeconds(10))) - { - _logger.Warn("Upgradeable read lock timed out"); - _bufferListLock.EnterUpgradeableReadLock(); - } - } - - /// - /// Releases the upgradeable read lock on the buffer list, allowing other threads to acquire exclusive or read - /// access. - /// - /// - /// Call this method after completing operations that required an upgradeable read lock on the buffer list. Failing - /// to release the lock may result in deadlocks or reduced concurrency. - /// - private void ReleaseBufferListUpgradeableReadLock () - { - _bufferListLock.ExitUpgradeableReadLock(); - } - - /// - /// Upgrades the buffer list lock from a reader lock to a writer lock, waiting up to 10 seconds before forcing the - /// upgrade if necessary. - /// - /// - /// If the writer lock cannot be acquired within 10 seconds, the method logs a warning and then blocks until the - /// writer lock is obtained. Call this method only when the current thread already holds a reader lock on the buffer - /// list. - /// - private void UpgradeBufferlistLockToWriterLock () - { - if (!_bufferListLock.TryEnterWriteLock(TimeSpan.FromSeconds(10))) - { - _logger.Warn("Writer lock upgrade timed out"); - _bufferListLock.EnterWriteLock(); - } - } - - /// - /// Downgrades the buffer list lock from write mode to allow other threads to acquire read access. - /// - /// - /// Call this method after completing write operations to permit concurrent read access to the buffer list. The - /// calling thread must hold the write lock before invoking this method. - /// - private void DowngradeBufferListLockFromWriterLock () - { - _bufferListLock.ExitWriteLock(); - } - #if DEBUG /// /// Outputs detailed information about the specified log buffer to the trace logger for debugging purposes. @@ -2395,7 +1863,7 @@ private static void DumpBufferInfos (LogBuffer buffer) public void Dispose () { Dispose(true); - GC.SuppressFinalize(this); // Suppress finalization (not needed but best practice) + GC.SuppressFinalize(this); } /// @@ -2415,10 +1883,15 @@ protected virtual void Dispose (bool disposing) { if (disposing) { + //Keep Dispose Order unless otherwise noted. + //For example, the progress reporter waits 2 seconds for the dispatch task + //and DeleteAllContent may trigger final events. DeleteAllContent(); _cts.Dispose(); - _lastBufferIndex.Dispose(); + BufferIndex.Dispose(); + _progressReporter.Dispose(); _mmfReader?.Dispose(); + } _disposed = true; @@ -2520,4 +1993,15 @@ protected virtual void OnRespawned () } #endregion Event Handlers + + #region IBufferPinning + + /// + PinHandle IBufferPinning.PinRange (int startLine, int endLine) + { + using var readLock = BufferIndex.AcquireReadLock(); + return BufferIndex.PinRange(startLine, endLine); + } + + #endregion } \ No newline at end of file diff --git a/src/LogExpert.Core/Classes/Log/ProgressReporters/NullProgressReporter.cs b/src/LogExpert.Core/Classes/Log/ProgressReporters/NullProgressReporter.cs new file mode 100644 index 00000000..7388f6fe --- /dev/null +++ b/src/LogExpert.Core/Classes/Log/ProgressReporters/NullProgressReporter.cs @@ -0,0 +1,23 @@ +using LogExpert.Core.Interfaces; + +namespace LogExpert.Core.Classes.Log.ProgressReporters; + +/// +/// No-op reporter for benchmarks and unit tests. Zero allocation, zero overhead. +/// +public sealed class NullProgressReporter : ILoadProgressReporter +{ + public static readonly NullProgressReporter Instance = new(); + + public void ReportProgress (string fileName, long position, long fileLength) { } + + public void ReportComplete (string fileName, long position, long fileLength) { } + + public void ReportNewFile (string fileName, long position, long fileLength) { } + + public void ReportLoadingStarted (string fileName) { } + + public void ReportLoadingFinished () { } + + public void Dispose () { } +} \ No newline at end of file diff --git a/src/LogExpert.Core/Classes/Log/ProgressReporters/PeriodicProgressReporter.cs b/src/LogExpert.Core/Classes/Log/ProgressReporters/PeriodicProgressReporter.cs new file mode 100644 index 00000000..bd5735d9 --- /dev/null +++ b/src/LogExpert.Core/Classes/Log/ProgressReporters/PeriodicProgressReporter.cs @@ -0,0 +1,129 @@ +using LogExpert.Core.EventArguments; +using LogExpert.Core.Interfaces; + +namespace LogExpert.Core.Classes.Log.ProgressReporters; + +/// +/// Periodically dispatches coalesced progress events on a background thread. +/// The I/O thread never blocks on subscribers — it writes volatile state and returns. +/// +internal sealed class PeriodicProgressReporter : ILoadProgressReporter +{ + private readonly Task _dispatchTask; + private readonly CancellationTokenSource _cts = new(); + private readonly TimeSpan _dispatchInterval; + + public event EventHandler? LoadFile; + public event EventHandler? LoadingStarted; + public event EventHandler? LoadingFinished; + + // Volatile state: written by I/O thread, read by dispatch loop + private volatile ProgressState _latestProgress = ProgressState.Empty; + private volatile ProgressState _latestComplete = ProgressState.Empty; + + private volatile bool _hasProgress; + private volatile bool _hasComplete; + private volatile bool _hasNewFile; + private volatile bool _hasStarted; + private volatile bool _hasFinished; + + public PeriodicProgressReporter (TimeSpan? dispatchInterval = null) + { + _dispatchInterval = dispatchInterval ?? TimeSpan.FromMilliseconds(200); + _dispatchTask = Task.Run(DispatchLoop); + } + + // I/O thread calls (non-blocking) + public void ReportProgress (string fileName, long position, long fileLength) + { + _latestProgress = new ProgressState(fileName, position, fileLength); + _hasProgress = true; + } + + public void ReportComplete (string fileName, long position, long fileLength) + { + _latestComplete = new ProgressState(fileName, position, fileLength); + _hasComplete = true; + } + + public void ReportNewFile (string fileName, long position, long fileLength) + { + _latestComplete = new ProgressState(fileName, position, fileLength); + _hasNewFile = true; + } + + public void ReportLoadingStarted (string fileName) + { + _latestProgress = new ProgressState(fileName, 0, 0); + _hasStarted = true; + } + + public void ReportLoadingFinished () + { + _hasFinished = true; + } + + // Dispatch loop (fires in lifecycle order: Started → Progress → NewFile → Complete → Finished) + private async Task DispatchLoop () + { + var token = _cts.Token; + + while (!token.IsCancellationRequested) + { + try + { + await Task.Delay(_dispatchInterval, token).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + break; + } + + if (_hasStarted) + { + _hasStarted = false; + var s = _latestProgress; + LoadingStarted?.Invoke(this, new LoadFileEventArgs(s.FileName, 0, false, 0, false)); + } + + if (_hasProgress) + { + _hasProgress = false; + var s = _latestProgress; + LoadFile?.Invoke(this, new LoadFileEventArgs(s.FileName, s.Position, false, s.FileLength, false)); + } + + if (_hasNewFile) + { + _hasNewFile = false; + var s = _latestComplete; + LoadFile?.Invoke(this, new LoadFileEventArgs(s.FileName, s.Position, false, s.FileLength, true)); + } + + if (_hasComplete) + { + _hasComplete = false; + var s = _latestComplete; + LoadFile?.Invoke(this, new LoadFileEventArgs(s.FileName, s.Position, true, s.FileLength, false)); + } + + if (_hasFinished) + { + _hasFinished = false; + LoadingFinished?.Invoke(this, EventArgs.Empty); + } + } + } + + private sealed record ProgressState (string FileName, long Position, long FileLength) + { + public static readonly ProgressState Empty = new(string.Empty, 0, 0); + } + + public void Dispose () + { + _cts.Cancel(); + _ = _dispatchTask.Wait(TimeSpan.FromSeconds(2)); + _cts.Dispose(); + } +} \ No newline at end of file diff --git a/src/LogExpert.Core/Classes/Log/PositionAwareStreamReaderBase.cs b/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderBase.cs similarity index 99% rename from src/LogExpert.Core/Classes/Log/PositionAwareStreamReaderBase.cs rename to src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderBase.cs index 666a8606..e186bc00 100644 --- a/src/LogExpert.Core/Classes/Log/PositionAwareStreamReaderBase.cs +++ b/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderBase.cs @@ -2,7 +2,7 @@ using LogExpert.Core.Entities; -namespace LogExpert.Core.Classes.Log; +namespace LogExpert.Core.Classes.Log.Streamreaders; public abstract class PositionAwareStreamReaderBase : LogStreamReaderBase { diff --git a/src/LogExpert.Core/Classes/Log/PositionAwareStreamReaderChannel.cs b/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderChannel.cs similarity index 99% rename from src/LogExpert.Core/Classes/Log/PositionAwareStreamReaderChannel.cs rename to src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderChannel.cs index 1acc7120..8edc6cf8 100644 --- a/src/LogExpert.Core/Classes/Log/PositionAwareStreamReaderChannel.cs +++ b/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderChannel.cs @@ -7,7 +7,7 @@ using LogExpert.Core.Entities; using LogExpert.Core.Interfaces; -namespace LogExpert.Core.Classes.Log; +namespace LogExpert.Core.Classes.Log.Streamreaders; public class PositionAwareStreamReaderChannel : LogStreamReaderBase, ILogStreamReaderMemory { diff --git a/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderDirect.cs b/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderDirect.cs new file mode 100644 index 00000000..aef38bb2 --- /dev/null +++ b/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderDirect.cs @@ -0,0 +1,291 @@ +using System.Buffers; +using System.Text; + +using LogExpert.Core.Classes.Log.Buffers; +using LogExpert.Core.Entities; +using LogExpert.Core.Interfaces; + +namespace LogExpert.Core.Classes.Log.Streamreaders; + +/// +/// Reads decoded characters directly into pooled char[] blocks via StreamReader.Read(), +/// scans for newline boundaries, and returns ReadOnlyMemory<char> slices without +/// allocating per-line strings. Eliminates the StreamReader.ReadLine() string allocation. +/// +public class PositionAwareStreamReaderDirect : PositionAwareStreamReaderBase, ILogStreamReaderMemory +{ + #region Constants + + private const int BLOCK_SIZE = 32_768; // 64 KB (32K chars × 2 bytes), under LOH threshold + private const char CHAR_LF = '\n'; + private const char CHAR_CR = '\r'; + + #endregion + + #region Fields + + private char[] _readBlock; + private int _readBlockLength; // valid chars in _readBlock + private int _scanOffset; // current scan position in _readBlock + private bool _eof; + private int _newLineSequenceLength; + private readonly List _completedBlocks = []; + + public override bool IsDisposed { get; protected set; } + + #endregion + + #region cTor + + public PositionAwareStreamReaderDirect (Stream stream, EncodingOptions encodingOptions, int maximumLineLength) + : base(stream, encodingOptions, maximumLineLength) + { + _readBlock = ArrayPool.Shared.Rent(BLOCK_SIZE); + _readBlockLength = 0; + _scanOffset = 0; + _eof = false; + } + + #endregion + + #region Public methods + + public override string ReadLine () + { + return TryReadLine(out var memory) ? memory.ToString() : null; + } + + /// + /// Reads the next line by scanning the current block for '\n'. If the block is exhausted, + /// tail-copies the partial line to a new block and refills. Returns a zero-copy + /// ReadOnlyMemory<char> slice into the pooled block. + /// + public bool TryReadLine (out ReadOnlyMemory lineMemory) + { + var reader = GetStreamReader(); + + if (_newLineSequenceLength == 0) + { + _newLineSequenceLength = GuessNewLineSequenceLength(reader); + } + + while (true) + { + // If we have data to scan, look for \n + if (_scanOffset < _readBlockLength) + { + var searchSpan = _readBlock.AsSpan(_scanOffset, _readBlockLength - _scanOffset); + var lfIndex = searchSpan.IndexOf(CHAR_LF); + + if (lfIndex >= 0) + { + // Found a line boundary + var lineLength = lfIndex; + + // Strip \r if present before \n + if (lineLength > 0 && searchSpan[lineLength - 1] == CHAR_CR) + { + lineLength--; + } + + // Enforce MaximumLineLength + var cappedLength = Math.Min(lineLength, MaximumLineLength); + + lineMemory = _readBlock.AsMemory(_scanOffset, cappedLength); + + // Update byte position: line chars + lineLength + var contentSpan = _readBlock.AsSpan(_scanOffset, lineLength); + MovePosition(Encoding.GetByteCount(contentSpan) + _newLineSequenceLength); + + // Advance scan past the \n + _scanOffset += lfIndex + 1; + + return true; + } + } + + // No \n found (or no data at all). Need to refill. + if (_eof) + { + // Emit remaining content as final line (no trailing newline) + if (_scanOffset < _readBlockLength) + { + var remaining = _readBlockLength - _scanOffset; + var cappedLength = Math.Min(remaining, MaximumLineLength); + lineMemory = _readBlock.AsMemory(_scanOffset, cappedLength); + + var fullSpan = _readBlock.AsSpan(_scanOffset, remaining); + MovePosition(Encoding.GetByteCount(fullSpan)); + + _scanOffset = _readBlockLength; + return true; + } + + lineMemory = default; + return false; + } + + // Tail-copy: move unconsumed chars to a new block and refill + RefillBlock(reader); + } + } + + public void ReturnMemory (ReadOnlyMemory memory) + { + // Bulk return via DetachBlocks()/Dispose(). Individual return not needed. + } + + /// + /// Gets the block allocator for compatibility with the DetachBlocks pattern. + /// This reader manages its own blocks directly rather than through CharBlockAllocator. + /// + public CharBlockAllocator? BlockAllocator => null; + + /// + /// Detaches completed blocks (fully scanned) for transfer to the LogBuffer. + /// The current _readBlock (partially scanned) stays with the reader. + /// + public List DetachBlocks () + { + // Nothing to detach: no completed blocks and no lines were scanned from the current block. + if (_completedBlocks.Count == 0 && _scanOffset == 0) + { + return []; + } + + // The current _readBlock contains memory backing lines already added to the LogBuffer. + // It must be transferred to the buffer along with any completed blocks. + _completedBlocks.Add(_readBlock); + + // Rent a fresh block and carry over any unscanned data (partial line in progress) + var tailLength = _readBlockLength - _scanOffset; + var newBlock = ArrayPool.Shared.Rent(BLOCK_SIZE); + + if (tailLength > 0) + { + _readBlock.AsSpan(_scanOffset, tailLength).CopyTo(newBlock.AsSpan(0, tailLength)); + } + + _readBlock = newBlock; + _readBlockLength = tailLength; + _scanOffset = 0; + + var blocks = _completedBlocks.ToList(); + _completedBlocks.Clear(); + return blocks; + } + + #endregion + + #region Private Methods + + private void RefillBlock (StreamReader reader) + { + var tailLength = _readBlockLength - _scanOffset; + + // Rent a new block + var newBlock = ArrayPool.Shared.Rent(BLOCK_SIZE); + + // Copy the tail (partial line) to the start of the new block + if (tailLength > 0) + { + _readBlock.AsSpan(_scanOffset, tailLength).CopyTo(newBlock.AsSpan(0, tailLength)); + } + + // The old block is fully scanned — add to completed list + _completedBlocks.Add(_readBlock); + + _readBlock = newBlock; + _scanOffset = 0; + + // Fill the rest of the block from the stream + var charsRead = reader.Read(newBlock, tailLength, BLOCK_SIZE - tailLength); + + _readBlockLength = tailLength + charsRead; + + if (charsRead == 0) + { + _eof = true; + } + } + + private int GuessNewLineSequenceLength (StreamReader reader) + { + var currentPos = Position; + + try + { + // Fill initial block + var charsRead = reader.Read(_readBlock, 0, BLOCK_SIZE); + _readBlockLength = charsRead; + _scanOffset = 0; + + if (charsRead == 0) + { + _eof = true; + return 0; + } + + // Find first \n to determine newline sequence + var span = _readBlock.AsSpan(0, _readBlockLength); + var lfIndex = span.IndexOf(CHAR_LF); + + if (lfIndex < 0) + { + // No newline found in first block — assume single-byte + return 1; + } + + if (lfIndex > 0 && span[lfIndex - 1] == CHAR_CR) + { + // \r\n + Span newline = ['\r', '\n']; + return Encoding.GetByteCount(newline); + } + + // \n only + Span singleLf = ['\n']; + return Encoding.GetByteCount(singleLf); + } + finally + { + // Reset position — the filled block data is kept, no re-read needed + // Position is reset to original so byte tracking starts clean + Position = currentPos; + + // Re-read the block since Position setter resets the stream + _scanOffset = 0; + var charsRead = reader.Read(_readBlock, 0, BLOCK_SIZE); + _readBlockLength = charsRead; + if (charsRead == 0) + { + _eof = true; + } + } + } + + protected override void Dispose (bool disposing) + { + if (disposing) + { + // Return the current (partially scanned) block + if (_readBlock != null) + { + ArrayPool.Shared.Return(_readBlock); + _readBlock = null!; + } + + // Return any completed blocks not yet detached + foreach (var block in _completedBlocks) + { + ArrayPool.Shared.Return(block); + } + + _completedBlocks.Clear(); + } + + base.Dispose(disposing); + } + + #endregion +} \ No newline at end of file diff --git a/src/LogExpert.Core/Classes/Log/PositionAwareStreamReaderLegacy.cs b/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderLegacy.cs similarity index 97% rename from src/LogExpert.Core/Classes/Log/PositionAwareStreamReaderLegacy.cs rename to src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderLegacy.cs index b30baefe..033de390 100644 --- a/src/LogExpert.Core/Classes/Log/PositionAwareStreamReaderLegacy.cs +++ b/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderLegacy.cs @@ -1,6 +1,6 @@ using LogExpert.Core.Entities; -namespace LogExpert.Core.Classes.Log; +namespace LogExpert.Core.Classes.Log.Streamreaders; public class PositionAwareStreamReaderLegacy (Stream stream, EncodingOptions encodingOptions, int maximumLineLength) : PositionAwareStreamReaderBase(stream, encodingOptions, maximumLineLength) { diff --git a/src/LogExpert.Core/Classes/Log/PositionAwareStreamReaderPipeline.cs b/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderPipeline.cs similarity index 99% rename from src/LogExpert.Core/Classes/Log/PositionAwareStreamReaderPipeline.cs rename to src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderPipeline.cs index deff3ff1..b85fd82f 100644 --- a/src/LogExpert.Core/Classes/Log/PositionAwareStreamReaderPipeline.cs +++ b/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderPipeline.cs @@ -6,7 +6,7 @@ using LogExpert.Core.Entities; using LogExpert.Core.Interfaces; -namespace LogExpert.Core.Classes.Log; +namespace LogExpert.Core.Classes.Log.Streamreaders; public class PositionAwareStreamReaderPipeline : LogStreamReaderBase, ILogStreamReaderMemory { diff --git a/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderPipelineNew.cs b/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderPipelineNew.cs new file mode 100644 index 00000000..e35b5951 --- /dev/null +++ b/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderPipelineNew.cs @@ -0,0 +1,702 @@ +using System.Buffers; +using System.Collections.Concurrent; +using System.IO.Pipelines; +using System.Text; + +using LogExpert.Core.Entities; +using LogExpert.Core.Interfaces; + +namespace LogExpert.Core.Classes.Log.Streamreaders; + +/// +/// EXPERIMENTAL: TypedPipeline-based reader for benchmarking comparison. +/// Uses multi-threaded pipeline with BlockingCollection stages. +/// Expected to be 15-25% slower than PositionAwareStreamReaderPipeline due to pipeline overhead. +/// +public class PositionAwareStreamReaderPipelineNew : LogStreamReaderBase, ILogStreamReaderMemory +{ + private const int DEFAULT_BYTE_BUFFER_SIZE = 64 * 1024; // 64 KB + private const int MINIMUM_READ_AHEAD_SIZE = 4 * 1024; // 4 KB + private const int DEFAULT_CHANNEL_CAPACITY = 128; // Number of line segments + + private static readonly Encoding[] _preambleEncodings = + [ + Encoding.UTF8, + Encoding.Unicode, + Encoding.BigEndianUnicode, + Encoding.UTF32 + ]; + + private readonly StreamPipeReaderOptions _streamPipeReaderOptions = new(bufferSize: DEFAULT_BYTE_BUFFER_SIZE, minimumReadSize: MINIMUM_READ_AHEAD_SIZE, leaveOpen: true); + private readonly int _maximumLineLength; + private readonly Lock _reconfigureLock = new(); + private readonly Stream _stream; + private readonly Encoding _encoding; + private readonly int _charBufferSize; + private readonly long _preambleLength; + + private LineSegment? _currentSegment; + private PipeReader _pipeReader; + private CancellationTokenSource _cts; + private Task _producerTask; + private bool _isDisposed; + private long _position; + private BlockingCollection _lineQueue; + private Exception _producerException; + private IPipeline _pipeline; + + public PositionAwareStreamReaderPipelineNew (Stream stream, EncodingOptions encodingOptions, int maximumLineLength) + { + ArgumentNullException.ThrowIfNull(stream); + + if (!stream.CanRead) + { + throw new ArgumentException("Stream must support reading.", nameof(stream)); + } + + if (!stream.CanSeek) + { + throw new ArgumentException("Stream must support seeking.", nameof(stream)); + } + + if (maximumLineLength <= 0) + { + maximumLineLength = 1024; + } + + _maximumLineLength = maximumLineLength; + var (length, detectedEncoding) = DetectPreambleLength(stream); + _preambleLength = length; + _encoding = DetermineEncoding(encodingOptions, detectedEncoding); + + _stream = stream; + _charBufferSize = Math.Max(_encoding.GetMaxCharCount(DEFAULT_BYTE_BUFFER_SIZE), _maximumLineLength + 2); + + RestartPipelineInternal(0); + } + + public override long Position + { + get => Interlocked.Read(ref _position); + set + { + ArgumentOutOfRangeException.ThrowIfNegative(value); + RestartPipeline(value); + } + } + + public override bool IsBufferComplete => true; + + public override Encoding Encoding => _encoding; + + public override bool IsDisposed + { + get => _isDisposed; + protected set => _isDisposed = value; + } + + public override int ReadChar () + { + throw new NotSupportedException("PipelineLogStreamReader currently supports line-based reads only."); + } + + public override string ReadLine () + { + if (TryReadLine(out var lineMemory)) + { + return new string(lineMemory.Span); + } + + return null; + } + + public bool TryReadLine (out ReadOnlyMemory lineMemory) + { + ObjectDisposedException.ThrowIf(IsDisposed, GetType()); + + var producerEx = Volatile.Read(ref _producerException); + if (producerEx != null) + { + throw new InvalidOperationException("Producer task encountered an error.", producerEx); + } + + var queue = _lineQueue; + var cts = _cts; + + if (queue == null || cts == null) + { + lineMemory = default; + return false; + } + + try + { + // With pre-filled queue, data should be available immediately + if (!queue.TryTake(out var segment, 50, cts.Token)) + { + lineMemory = default; + return false; + } + + _currentSegment?.Dispose(); + _currentSegment = segment; + + if (segment.IsEof) + { + lineMemory = default; + return false; + } + + lineMemory = new ReadOnlyMemory(segment.Buffer, 0, segment.Length); + _ = Interlocked.Exchange(ref _position, segment.ByteOffset + segment.ByteLength); + return true; + } + catch (OperationCanceledException) + { + lineMemory = default; + return false; + } + catch (ObjectDisposedException) + { + lineMemory = default; + return false; + } + } + + public void ReturnMemory (ReadOnlyMemory memory) + { + // No-op for this implementation + } + + protected override void Dispose (bool disposing) + { + if (_isDisposed) + { + return; + } + + if (disposing) + { + using (_reconfigureLock.EnterScope()) + { + CancelPipelineLocked(); + + if (_lineQueue != null) + { + while (_lineQueue.TryTake(out var segment)) + { + segment.Dispose(); + } + + _lineQueue.Dispose(); + } + + _stream?.Dispose(); + } + } + + _isDisposed = true; + } + + private void RestartPipelineInternal (long startPosition) + { + _ = _stream.Seek(_preambleLength + startPosition, SeekOrigin.Begin); + + _pipeReader = PipeReader.Create(_stream, _streamPipeReaderOptions); + _lineQueue = new BlockingCollection(new ConcurrentQueue(), DEFAULT_CHANNEL_CAPACITY); + + Volatile.Write(ref _producerException, null); + + // KEY FIX: Read and enqueue first line synchronously to guarantee immediate data availability + var firstLine = ReadFirstLineSynchronously(startPosition); + long nextByteOffset = startPosition; + + if (firstLine.HasValue) + { + _lineQueue.Add(firstLine.Value); + nextByteOffset = firstLine.Value.ByteOffset + firstLine.Value.ByteLength; + } + + _cts = new CancellationTokenSource(); + + // Build TypedPipeline + var builder = new TypedPipelineBuilder(); + _pipeline = builder.AddStep(ProcessBuffer).Build(); + + _pipeline.Finished += segment => + { + if (segment.Buffer != null || segment.IsEof) + { + EnqueueLine(segment); + } + }; + + _producerTask = Task.Run(() => ProduceAsync(nextByteOffset, _cts.Token), CancellationToken.None); + + _ = Interlocked.Exchange(ref _position, startPosition); + } + + /// + /// Reads the first line from the stream synchronously to pre-fill the queue. + /// This ensures data is immediately available when TryReadLine() is called after a seek. + /// + private LineSegment? ReadFirstLineSynchronously (long startPosition) + { + const int FIRST_LINE_BUFFER_SIZE = 4096; + + var byteBuffer = ArrayPool.Shared.Rent(FIRST_LINE_BUFFER_SIZE); + var charBuffer = ArrayPool.Shared.Rent(_charBufferSize); + + try + { + var bytesRead = _stream.Read(byteBuffer, 0, FIRST_LINE_BUFFER_SIZE); + + if (bytesRead == 0) + { + return LineSegment.CreateEof(startPosition); + } + + var decoder = _encoding.GetDecoder(); + var charsDecoded = decoder.GetChars(byteBuffer, 0, bytesRead, charBuffer, 0, flush: false); + + var (newlineIndex, newlineChars) = FindNewlineIndex(charBuffer, 0, charsDecoded, false); + + if (newlineIndex == -1) + { + return null; + } + + var segment = CreateSegment(charBuffer, 0, newlineIndex, newlineChars, startPosition); + + return segment; + } + catch (Exception ex) when (ex is IOException or + DecoderFallbackException or + ObjectDisposedException) + { + return null; + } + finally + { + ArrayPool.Shared.Return(byteBuffer); + ArrayPool.Shared.Return(charBuffer); + } + } + + private void RestartPipeline (long newPosition) + { + using (_reconfigureLock.EnterScope()) + { + CancelPipelineLocked(); + RestartPipelineInternal(newPosition); + } + } + + private void CancelPipelineLocked () + { + if (_cts == null) + { + return; + } + + try + { + _cts.Cancel(); + } + catch (ObjectDisposedException) + { + // Ignore + } + + try + { + _producerTask?.Wait(); + } + catch (AggregateException ex) when (ex.InnerExceptions.All(e => e is OperationCanceledException)) + { + // Expected + } + finally + { + _pipeline?.Complete(); + _cts.Dispose(); + _cts = null; + } + + if (_lineQueue != null && !_lineQueue.IsAddingCompleted) + { + _lineQueue.CompleteAdding(); + } + + if (_pipeReader != null) + { + try + { + _pipeReader.Complete(); + } + catch (ObjectDisposedException) + { + // Ignore: shutdown race + } + catch (InvalidOperationException) + { + // Ignore: already completed or invalid teardown state + } + } + } + + private async Task ProduceAsync (long startByteOffset, CancellationToken token) + { + var charPool = ArrayPool.Shared; + char[] charBuffer = null; + Decoder decoder = null; + + try + { + charBuffer = charPool.Rent(_charBufferSize); + decoder = _encoding.GetDecoder(); + + var charsInBuffer = 0; + var byteOffset = startByteOffset; + + while (!token.IsCancellationRequested) + { + ReadResult result = await _pipeReader.ReadAsync(token).ConfigureAwait(false); + ReadOnlySequence buffer = result.Buffer; + + if (buffer.Length > 0) + { + // Create buffer data and feed to pipeline + var bufferData = new BufferData(buffer, charBuffer, charsInBuffer, decoder, byteOffset, result.IsCompleted); + + // Process and extract lines + var processResult = ProcessBufferAndExtractLines(bufferData); + + // Update state + charsInBuffer = processResult.RemainingChars; + byteOffset = processResult.NewByteOffset; + + var pipeline = _pipeline ?? throw new InvalidOperationException("Pipeline is not initialized."); + + // Feed each line to pipeline (which will enqueue them) + foreach (var segment in processResult.Lines) + { + + pipeline.Execute(new BufferData(default, segment.Buffer, 0, null, segment.ByteOffset, false) + { + PreExtractedSegment = segment + }); + } + + _pipeReader.AdvanceTo(buffer.End); + } + + if (result.IsCompleted) + { + if (charsInBuffer > 0) + { + var segment = CreateSegment(charBuffer, 0, charsInBuffer, 0, byteOffset); + EnqueueLine(segment); + byteOffset += segment.ByteLength; + } + + EnqueueLine(LineSegment.CreateEof(byteOffset)); + break; + } + } + } + catch (OperationCanceledException) + { + // Expected + } + catch (DecoderFallbackException ex) + { + Volatile.Write(ref _producerException, ex); + } + catch (ObjectDisposedException ex) + { + Volatile.Write(ref _producerException, ex); + } + catch (InvalidOperationException ex) + { + Volatile.Write(ref _producerException, ex); + } + catch + { + throw; + } + finally + { + _pipeline?.Complete(); + + try + { + _lineQueue?.CompleteAdding(); + } + catch (ObjectDisposedException) + { + // Ignore + } + + if (charBuffer != null) + { + charPool.Return(charBuffer); + } + } + } + + private LineSegment ProcessBuffer (BufferData bufferData) + { + // If pre-extracted, just return it (simulating pipeline overhead) + if (bufferData.PreExtractedSegment.HasValue) + { + return bufferData.PreExtractedSegment.Value; + } + + // This shouldn't happen in normal operation, but handle gracefully + // Return an empty line segment + return new LineSegment(null, 0, bufferData.ByteOffset, 0, false, false); + } + + private ProcessResult ProcessBufferAndExtractLines (BufferData bufferData) + { + var lines = new List(); + var localCharsInBuffer = bufferData.CharsInBuffer; + var localByteOffset = bufferData.ByteOffset; + + // Decode bytes to chars + if (bufferData.Buffer.IsSingleSegment) + { + var span = bufferData.Buffer.FirstSpan; + var charsAvailable = _charBufferSize - localCharsInBuffer; + + if (charsAvailable > 10) + { + bufferData.Decoder.Convert( + span, + bufferData.CharBuffer.AsSpan(localCharsInBuffer), + bufferData.IsCompleted, + out var usedBytes, + out var charsProduced, + out _); + + localCharsInBuffer += charsProduced; + localByteOffset += usedBytes; + } + } + + // Extract lines + var searchIndex = 0; + while (true) + { + var (newlineIndex, newlineChars) = FindNewlineIndex(bufferData.CharBuffer, searchIndex, localCharsInBuffer - searchIndex, false); + + if (newlineIndex == -1) + { + break; + } + + var lineLength = newlineIndex - searchIndex; + var segment = CreateSegment(bufferData.CharBuffer, searchIndex, lineLength, newlineChars, localByteOffset); + lines.Add(segment); + localByteOffset += segment.ByteLength; + searchIndex = newlineIndex + newlineChars; + } + + // Calculate remaining chars + var remaining = localCharsInBuffer - searchIndex; + if (remaining > 0 && searchIndex > 0) + { + bufferData.CharBuffer.AsSpan(searchIndex, remaining).CopyTo(bufferData.CharBuffer.AsSpan(0, remaining)); + } + + return new ProcessResult(lines, remaining, localByteOffset); + } + + private void EnqueueLine (LineSegment segment) + { + try + { + // Don't use cancellation token here - let the queue complete naturally + _lineQueue.Add(segment); + } + catch (InvalidOperationException) + { + // Collection was marked as complete, dispose the segment + segment.Dispose(); + } + } + + private static (int newLineIndex, int newLineChars) FindNewlineIndex ( + char[] buffer, + int start, + int available, + bool allowStandaloneCr) + { + var span = buffer.AsSpan(start, available); + + var lfIndex = span.IndexOf('\n'); + if (lfIndex != -1) + { + if (lfIndex > 0 && span[lfIndex - 1] == '\r') + { + return (newLineIndex: start + lfIndex - 1, newLineChars: 2); + } + + return (newLineIndex: start + lfIndex, newLineChars: 1); + } + + var crIndex = span.IndexOf('\r'); + if (crIndex != -1) + { + if (crIndex + 1 >= span.Length) + { + if (allowStandaloneCr) + { + return (newLineIndex: start + crIndex, newLineChars: 1); + } + + return (newLineIndex: -1, newLineChars: 0); + } + + if (span[crIndex + 1] != '\n') + { + return (newLineIndex: start + crIndex, newLineChars: 1); + } + } + + return (newLineIndex: -1, newLineChars: 0); + } + + private LineSegment CreateSegment ( + char[] source, + int start, + int lineLength, + int newlineChars, + long byteOffset) + { + var consumedChars = lineLength + newlineChars; + + var byteLength = consumedChars == 0 + ? 0 + : _encoding.GetByteCount(source, start, consumedChars); + + var logicalLength = Math.Min(lineLength, _maximumLineLength); + var truncated = lineLength > logicalLength; + + var rentalLength = Math.Max(logicalLength, 1); + var buffer = ArrayPool.Shared.Rent(rentalLength); + + if (logicalLength > 0) + { + source.AsSpan(start, logicalLength).CopyTo(buffer.AsSpan(0, logicalLength)); + } + + return new LineSegment(buffer, logicalLength, byteOffset, byteLength, truncated, false); + } + + // Pipeline data structures + private class BufferData + { + public ReadOnlySequence Buffer { get; } + public char[] CharBuffer { get; } + public int CharsInBuffer { get; } + public Decoder Decoder { get; } + public long ByteOffset { get; } + public bool IsCompleted { get; } + public LineSegment? PreExtractedSegment { get; set; } + + public BufferData (ReadOnlySequence buffer, char[] charBuffer, int charsInBuffer, Decoder decoder, long byteOffset, bool isCompleted) + { + Buffer = buffer; + CharBuffer = charBuffer; + CharsInBuffer = charsInBuffer; + Decoder = decoder; + ByteOffset = byteOffset; + IsCompleted = isCompleted; + } + } + + private record ProcessResult (List Lines, int RemainingChars, long NewByteOffset); + private record PipelineInput (ReadOnlySequence Buffer, char[] CharBuffer, int CharsInBuffer, Decoder Decoder, long ByteOffset, bool IsCompleted); + private record DecodeResult (char[] CharBuffer, int CharsInBuffer, long ByteOffset); + private record ExtractResult (List Lines); + + private readonly struct LineSegment : IDisposable + { + public char[] Buffer { get; } + public int Length { get; } + public long ByteOffset { get; } + public int ByteLength { get; } + public bool IsTruncated { get; } + public bool IsEof { get; } + + public LineSegment (char[] buffer, int length, long byteOffset, int byteLength, bool isTruncated, bool isEof) + { + Buffer = buffer; + Length = length; + ByteOffset = byteOffset; + ByteLength = byteLength; + IsTruncated = isTruncated; + IsEof = isEof; + } + + public void Dispose () + { + if (Buffer != null) + { + ArrayPool.Shared.Return(Buffer); + } + } + + public static LineSegment CreateEof (long byteOffset) + { + return new LineSegment(null, 0, byteOffset, 0, false, true); + } + } + + private static Encoding DetermineEncoding (EncodingOptions options, Encoding detectedEncoding) + { + return options?.Encoding != null + ? options.Encoding + : detectedEncoding ?? options?.DefaultEncoding ?? Encoding.Default; + } + + private static (int length, Encoding? detectedEncoding) DetectPreambleLength (Stream stream) + { + if (!stream.CanSeek) + { + return (0, null); + } + + var originalPos = stream.Position; + var buffer = new byte[4]; + _ = stream.Seek(0, SeekOrigin.Begin); + var readBytes = stream.Read(buffer, 0, buffer.Length); + _ = stream.Seek(originalPos, SeekOrigin.Begin); + + if (readBytes >= 2) + { + foreach (var encoding in _preambleEncodings) + { + var preamble = encoding.GetPreamble(); + var fail = false; + for (var i = 0; i < readBytes && i < preamble.Length; ++i) + { + if (buffer[i] != preamble[i]) + { + fail = true; + break; + } + } + + if (!fail) + { + return (preamble.Length, encoding); + } + } + } + + return (0, null); + } +} diff --git a/src/LogExpert.Core/Classes/Log/PositionAwareStreamReaderSystem.cs b/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderSystem.cs similarity index 70% rename from src/LogExpert.Core/Classes/Log/PositionAwareStreamReaderSystem.cs rename to src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderSystem.cs index 41958b4d..fce09075 100644 --- a/src/LogExpert.Core/Classes/Log/PositionAwareStreamReaderSystem.cs +++ b/src/LogExpert.Core/Classes/Log/Streamreaders/PositionAwareStreamReaderSystem.cs @@ -1,9 +1,10 @@ using System.Text; +using LogExpert.Core.Classes.Log.Buffers; using LogExpert.Core.Entities; using LogExpert.Core.Interfaces; -namespace LogExpert.Core.Classes.Log; +namespace LogExpert.Core.Classes.Log.Streamreaders; /// /// This class is responsible for reading line from the log file. It also decodes characters with the appropriate charset encoding. @@ -32,6 +33,20 @@ public PositionAwareStreamReaderSystem (Stream stream, EncodingOptions encodingO #endregion + #region Properties + + /// + /// Gets or creates the block allocator used by this reader instance. + /// The caller can detach the blocks after reading a buffer's worth of lines. + /// + public CharBlockAllocator BlockAllocator + { + get => field ??= new CharBlockAllocator(); + private set; + } + + #endregion + #region Public methods public override string ReadLine () @@ -73,30 +88,32 @@ public bool TryReadLine (out ReadOnlyMemory lineMemory) var line = reader.ReadLine(); - if (line != null) + if (line is null) { - MovePosition(Encoding.GetByteCount(line) + _newLineSequenceLength); + lineMemory = default; + return false; + } - if (line.Length > MaximumLineLength) - { - line = line[..MaximumLineLength]; - } + MovePosition(Encoding.GetByteCount(line) + _newLineSequenceLength); - // Store line for Memory access - lineMemory = line.AsMemory(); - return true; - } + var length = Math.Min(line.Length, MaximumLineLength); - lineMemory = default; - return false; + // Allocate from block and copy + var allocator = BlockAllocator; + var target = allocator.Rent(length); + line.AsSpan(0, length).CopyTo(target.Span); + lineMemory = target; + return true; } /// - /// Returns the memory buffer. For System reader, this is a no-op since we use string-backed Memory. + /// Returns the memory buffer. For the block-based reader, individual returns are not tracked — blocks are returned + /// in bulk via the BlockAllocator when the LogBuffer is evicted or the reader is disposed. /// public void ReturnMemory (ReadOnlyMemory memory) { - // No-op for System reader - string is already managed by GC + // Bulk return via BlockAllocator.DetachBlocks() or Dispose(). + // Individual per-line return is not needed with block-based allocation. } #endregion @@ -140,5 +157,16 @@ private int GuessNewLineSequenceLength (StreamReader reader) } } + protected override void Dispose (bool disposing) + { + if (disposing) + { + BlockAllocator?.Dispose(); + BlockAllocator = null; + } + + base.Dispose(disposing); + } + #endregion } \ No newline at end of file diff --git a/src/LogExpert.Core/Config/Preferences.cs b/src/LogExpert.Core/Config/Preferences.cs index a63e190c..37881e49 100644 --- a/src/LogExpert.Core/Config/Preferences.cs +++ b/src/LogExpert.Core/Config/Preferences.cs @@ -54,8 +54,8 @@ public List HilightGroupList /// /// /// This property controls line truncation at the I/O reader level before lines are processed by columnizers. - /// It is implemented in and - /// . Related property: + /// It is implemented in and + /// . Related property: /// controls display-level truncation in UI columns, which must not exceed this /// value. Default is 20000 characters. /// @@ -79,7 +79,7 @@ public List HilightGroupList [Newtonsoft.Json.JsonIgnore] public bool UseLegacyReader { get; set; } - public ReaderType ReaderType { get; set; } = ReaderType.Pipeline; + public ReaderType ReaderType { get; set; } = ReaderType.System; public List ToolEntries { get; set; } = []; diff --git a/src/LogExpert.Core/Entities/DefaultLogfileColumnizer.cs b/src/LogExpert.Core/Entities/DefaultLogfileColumnizer.cs index c64a8417..52750983 100644 --- a/src/LogExpert.Core/Entities/DefaultLogfileColumnizer.cs +++ b/src/LogExpert.Core/Entities/DefaultLogfileColumnizer.cs @@ -65,6 +65,11 @@ public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, st // No special handling needed for default columnizer } + public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, ReadOnlyMemory oldValue) + { + // No special handling needed for default columnizer + } + public string Text => GetName(); public static Priority GetPriority (string fileName, IEnumerable samples) diff --git a/src/LogExpert.Core/Enums/ReaderType.cs b/src/LogExpert.Core/Enums/ReaderType.cs index 65d89a82..dbfabfb0 100644 --- a/src/LogExpert.Core/Enums/ReaderType.cs +++ b/src/LogExpert.Core/Enums/ReaderType.cs @@ -18,5 +18,11 @@ public enum ReaderType /// /// System.IO.StreamReader based implementation. /// - System + System, + + /// + /// Direct-read implementation: reads decoded chars directly into pooled blocks via + /// StreamReader.Read(char[], offset, count), eliminating per-line string allocation. + /// + SystemDirect } diff --git a/src/LogExpert.Core/Interfaces/IBufferPinning.cs b/src/LogExpert.Core/Interfaces/IBufferPinning.cs new file mode 100644 index 00000000..15d8bbb3 --- /dev/null +++ b/src/LogExpert.Core/Interfaces/IBufferPinning.cs @@ -0,0 +1,27 @@ +using LogExpert.Core.Classes.Log.Buffers; + +namespace LogExpert.Core.Interfaces; + +/// +/// Capability interface for pinning buffer ranges so the GC-eviction thread cannot +/// reclaim them during a paint cycle. Used by ColumnCache for flicker-free rendering. +/// +/// Invariants: +/// +/// The returned keeps buffers alive until disposed. +/// Callers MUST dispose the handle — leaked handles prevent buffer eviction. +/// Pin-before-read: call BEFORE reading lines +/// to prevent the GC thread from evicting between pin and read. +/// +/// +public interface IBufferPinning +{ + /// + /// Pins all buffers that cover the line range [, ]. + /// Internally acquires and releases the necessary read lock. + /// + /// First line of the range to pin (inclusive). + /// Last line of the range to pin (inclusive). + /// A that must be disposed when the pinned range is no longer needed. + PinHandle PinRange(int startLine, int endLine); +} diff --git a/src/LogExpert.Core/Interfaces/ILoadProgressReporter.cs b/src/LogExpert.Core/Interfaces/ILoadProgressReporter.cs new file mode 100644 index 00000000..4c82d4dc --- /dev/null +++ b/src/LogExpert.Core/Interfaces/ILoadProgressReporter.cs @@ -0,0 +1,35 @@ +namespace LogExpert.Core.Interfaces; + +/// +/// Decouples file loading progress reporting from event dispatch. +/// The I/O path calls Report*() methods; the implementation decides +/// when and how to fire events. +/// +public interface ILoadProgressReporter : IDisposable +{ + /// + /// Report intermediate loading progress. Called from I/O thread. + /// Implementation may batch/coalesce/drop these. + /// + void ReportProgress (string fileName, long position, long fileLength); + + /// + /// Report that loading of a file segment is complete. + /// + void ReportComplete (string fileName, long position, long fileLength); + + /// + /// Report that a new file was detected (rollover). + /// + void ReportNewFile (string fileName, long position, long fileLength); + + /// + /// Report that a loading operation has started. + /// + void ReportLoadingStarted (string fileName); + + /// + /// Report that a loading operation has finished. + /// + void ReportLoadingFinished (); +} \ No newline at end of file diff --git a/src/LogExpert.Core/Interfaces/ILogWindow.cs b/src/LogExpert.Core/Interfaces/ILogWindow.cs index 6fb6386f..6c297b9e 100644 --- a/src/LogExpert.Core/Interfaces/ILogWindow.cs +++ b/src/LogExpert.Core/Interfaces/ILogWindow.cs @@ -1,6 +1,5 @@ using ColumnizerLib; -using LogExpert.Core.Classes.Log; using LogExpert.Core.Classes.Persister; namespace LogExpert.Core.Interfaces; @@ -172,18 +171,21 @@ public interface ILogWindow void Activate (); /// - /// Gets the instance that provides access to the + /// Gets the instance that provides access to the /// underlying log file content. /// /// - /// The that manages file access, buffering, and + /// The reader that manages file access, buffering, and /// multi-file coordination for this log window. /// /// - /// The LogfileReader handles all file I/O operations, including reading lines, + /// The Logfile Reader handles all file I/O operations, including reading lines, /// monitoring for file changes, and managing the buffer cache. + /// Query for capability interfaces (, + /// , ) + /// when specialized operations are needed. /// - LogfileReader LogFileReader { get; } + ILogfileReader LogFileReader { get; } /// /// Gets the text content of the currently selected cell or line. diff --git a/src/LogExpert.Core/Interfaces/ILogfileReader.cs b/src/LogExpert.Core/Interfaces/ILogfileReader.cs new file mode 100644 index 00000000..dc61baa8 --- /dev/null +++ b/src/LogExpert.Core/Interfaces/ILogfileReader.cs @@ -0,0 +1,183 @@ +using System.Text; + +using ColumnizerLib; + +using LogExpert.Core.Entities; +using LogExpert.Core.EventArguments; + +namespace LogExpert.Core.Interfaces; + +/// +/// Core seam for the Logfile Reader module — covers the 80% case: +/// reading Log Lines, getting the line count, reacting to file changes, and lifecycle. +/// +/// Adapters: the file-backed LogfileReader (production) and an +/// in-memory adapter for tests. Future adapters may support network streams or +/// compressed files. +/// +/// Capability interfaces: specialized operations are on separate interfaces +/// that the production adapter also implements. Query with pattern matching when needed: +/// +/// — navigation across physical file boundaries +/// — XML mode, pre-process columnizer, encoding changes +/// — pin buffer ranges during paint cycles (ColumnCache) +/// — debug-only buffer logging +/// +/// +///

Invariants

+/// +/// Line numbers are zero-based and dense: [0 .. LineCount-1]. +/// is eventually-consistent with the underlying source. +/// After a event, it reflects the new state. +/// All read methods (, +/// , ) are safe +/// to call from any thread. The implementation handles internal synchronization. +/// Events are raised on background threads. Callers must marshal to the +/// UI thread if needed. +/// must be called exactly once after construction +/// to begin producing events. Calling it again after +/// is not supported — create a new instance instead. +/// +/// +///

Ordering

+/// +/// Construct → set configuration (via ) → +/// fires once (initial file read begins) +/// fires repeatedly during initial read with progress +/// fires once (initial read complete; LineCount is valid) +/// fires on each append/truncate detected during monitoring +/// or → Dispose +/// +/// +///

Error Modes

+/// +/// : the watched file was deleted or became inaccessible. +/// After this event, read methods return null for affected lines. +/// : a previously-deleted file reappeared (e.g. log rotation). +/// The reader reloads content and fires . +/// Read methods return null for out-of-range lines or when the backing store is +/// temporarily unavailable (e.g. network timeout). They never throw. +/// +/// +public interface ILogfileReader : IAutoLogLineMemoryColumnizerCallback, IDisposable +{ + #region Events + + /// + /// Raised when the file size and/or line count changes (content appended, truncated, or rotated). + /// This is the primary event driving Tail Mode updates in the UI. + /// + event EventHandler FileSizeChanged; + + /// + /// Raised repeatedly during initial file load with progress information. + /// + event EventHandler LoadFile; + + /// + /// Raised once when the initial file load begins. + /// + event EventHandler LoadingStarted; + + /// + /// Raised once when the initial file load completes. After this event, + /// reflects the full file content. + /// + event EventHandler LoadingFinished; + + /// + /// Raised when the monitored file is deleted or becomes inaccessible. + /// + event EventHandler FileNotFound; + + /// + /// Raised when a previously-missing file reappears (e.g. after log rotation). + /// + event EventHandler Respawned; + + #endregion + + #region Core Read Properties + + /// + /// Total number of Log Lines currently available. Thread-safe. + /// Zero before fires for the first time. + /// + int LineCount { get; } + + /// + /// Whether this reader spans multiple physical files (Multi-File Mode). + /// Immutable after construction. + /// + bool IsMultiFile { get; } + + /// + /// The character encoding currently in use. + /// + Encoding CurrentEncoding { get; } + + /// + /// Total size, in bytes, of the backing file(s). Updated on each monitoring cycle. + /// + long FileSize { get; } + + #endregion + + #region Line Access + + // GetLogLineMemory(int) is inherited from IAutoLogLineMemoryColumnizerCallback. + // It is the synchronous single-line read. Returns null for out-of-range or unavailable lines. + + /// + /// Reads a single Log Line with a bounded wait. If the backing store is slow + /// (network share, deleted file), returns null after an internal timeout rather + /// than blocking indefinitely. The implementation may enter a fast-fail mode + /// after a timeout to prevent GUI freezes. + /// + /// Zero-based line number. + /// The Log Line, or null if unavailable within the timeout. + Task GetLogLineMemoryWithWait(int lineNum); + + /// + /// Batch read for a contiguous range of Log Lines. Optimized for paint-cycle + /// prefetch: acquires locks once for the entire range. + /// + /// Zero-based starting line number. + /// Number of lines to read. The returned array may be shorter + /// if the end of the file is reached. Individual entries may be null if a buffer + /// is temporarily unavailable. + /// Array of Log Lines. Never null; may be empty or shorter than . + ILogLineMemory[] GetLogLineMemories(int startLine, int count); + + #endregion + + #region Lifecycle + + /// + /// Begins background monitoring for file changes (Tail Mode). Triggers initial file read, + /// producing // + /// events, then monitors for appends. + /// + void StartMonitoring(); + + /// + /// Synchronously stops background monitoring and releases file handles. Blocks until + /// background tasks complete (with a bounded timeout). After this call, no more events + /// will be raised. + /// + void StopMonitoring(); + + /// + /// Asynchronously stops background monitoring. Returns immediately; cleanup happens + /// on a background thread. Use when a fast-responding GUI is needed (e.g. closing a tab). + /// + void StopMonitoringAsync(); + + /// + /// Clears all buffered content and releases memory. Call only when the reader is about + /// to be disposed. After this call, all read methods return null/empty. + /// + void DeleteAllContent(); + + #endregion +} diff --git a/src/LogExpert.Core/Interfaces/ILogfileReaderConfiguration.cs b/src/LogExpert.Core/Interfaces/ILogfileReaderConfiguration.cs new file mode 100644 index 00000000..db45b49f --- /dev/null +++ b/src/LogExpert.Core/Interfaces/ILogfileReaderConfiguration.cs @@ -0,0 +1,47 @@ +using System.Text; + +using ColumnizerLib; + +namespace LogExpert.Core.Interfaces; + +/// +/// Capability interface for mutable configuration that affects how the Logfile Reader +/// parses lines. Typically set once at load time or when the user changes the Columnizer. +/// +/// Ordering: +/// +/// Set and together +/// before calling +/// or immediately after stopping monitoring for a Columnizer switch. +/// clears all buffers and triggers a full re-read +/// on the next monitoring cycle. +/// +/// +public interface ILogfileReaderConfiguration +{ + /// + /// Enables or disables XML log parsing mode. Must be set before + /// . Changing after monitoring + /// has started triggers a re-read of the file content. + /// + bool IsXmlMode { get; set; } + + /// + /// The XML log configuration (start/end tags) used when is true. + /// Must be set before if XML mode is enabled. + /// + IXmlLogConfiguration XmlLogConfig { get; set; } + + /// + /// Optional pre-process Columnizer that transforms each line as it is loaded from disk. + /// Set to null to disable pre-processing. Can be changed at any time; takes effect on + /// the next buffer read. + /// + IPreProcessColumnizerMemory PreProcessColumnizer { get; set; } + + /// + /// Changes the character encoding used to read the file. Invalidates all cached buffers; + /// the next access will re-read from disk with the new encoding. + /// + void ChangeEncoding(Encoding encoding); +} diff --git a/src/LogExpert.Core/Interfaces/ILogfileReaderDiagnostics.cs b/src/LogExpert.Core/Interfaces/ILogfileReaderDiagnostics.cs new file mode 100644 index 00000000..a2891d6b --- /dev/null +++ b/src/LogExpert.Core/Interfaces/ILogfileReaderDiagnostics.cs @@ -0,0 +1,22 @@ +namespace LogExpert.Core.Interfaces; + +/// +/// Capability interface for debug-only buffer diagnostics. +/// No caller should depend on this for correctness — it writes to the diagnostic log. +/// +public interface ILogfileReaderDiagnostics +{ +#if DEBUG + /// + /// Logs detailed buffer information for the given line to the diagnostic output. + /// Only functional in debug builds; no-op in release. + /// + void LogBufferInfoForLine(int lineNum); + + /// + /// Logs overall buffer and LRU cache diagnostics. + /// Only functional in debug builds; no-op in release. + /// + void LogBufferDiagnostic(); +#endif +} diff --git a/src/LogExpert.Core/Interfaces/IMultiFileNavigation.cs b/src/LogExpert.Core/Interfaces/IMultiFileNavigation.cs new file mode 100644 index 00000000..e0530692 --- /dev/null +++ b/src/LogExpert.Core/Interfaces/IMultiFileNavigation.cs @@ -0,0 +1,49 @@ +using ColumnizerLib; + +namespace LogExpert.Core.Interfaces; + +/// +/// Capability interface for navigation across physical file boundaries in Multi-File Mode. +/// Query for this interface only when is true. +/// +/// Invariants: +/// +/// All line numbers are zero-based virtual line numbers spanning the combined file set. +/// "Next/Prev" methods return -1 when no boundary exists in that direction. +/// For single-file readers that also implement this interface, methods return +/// sensible defaults (the only file name, -1 for next/prev). +/// +/// +public interface IMultiFileNavigation +{ + /// + /// Returns the file name (full path) of the physical file containing the given line. + /// For single-file readers, always returns the primary file name. + /// + string GetLogFileNameForLine(int lineNum); + + /// + /// Returns the for the physical file containing the given line. + /// Returns null if the line number is out of range. + /// + ILogFileInfo GetLogFileInfoForLine(int lineNum); + + /// + /// Returns the first line number of the next physical file after , + /// or -1 if is in the last file. + /// + int GetNextMultiFileLine(int lineNum); + + /// + /// Returns the first line number of the previous physical file before , + /// or -1 if is in the first file. + /// + int GetPrevMultiFileLine(int lineNum); + + /// + /// Maps a virtual (combined multi-file) line number to the real line number within + /// its physical file. Used when launching external tools that need file-relative positions. + /// Returns -1 if the mapping cannot be determined. + /// + int GetRealLineNumForVirtualLineNum(int lineNum); +} diff --git a/src/LogExpert.Tests/Buffers/BufferIndexSnapshotTests.cs b/src/LogExpert.Tests/Buffers/BufferIndexSnapshotTests.cs new file mode 100644 index 00000000..9b2100f3 --- /dev/null +++ b/src/LogExpert.Tests/Buffers/BufferIndexSnapshotTests.cs @@ -0,0 +1,47 @@ +using LogExpert.Core.Classes.Log.Buffers; + +using NUnit.Framework; + +namespace LogExpert.Tests.Buffers; + +[TestFixture] +internal class BufferIndexSnapshotTests +{ + [Test] + public void ToString_ReturnsFormattedSummary () + { + var snapshot = new BufferIndexSnapshot + { + BufferCount = 5, + TotalLineCount = 2500, + LruCacheCount = 3, + Buffers = [] + }; + + Assert.That(snapshot.ToString(), Is.EqualTo("Buffers=5, Lines=2500, LRU=3")); + } + + [Test] + public void BufferInfo_RecordEquality () + { + var a = new BufferIndexSnapshot.BufferInfo(0, 100, 0, 1000, false, "file.log"); + var b = new BufferIndexSnapshot.BufferInfo(0, 100, 0, 1000, false, "file.log"); + var c = new BufferIndexSnapshot.BufferInfo(100, 100, 1000, 1000, false, "file.log"); + + Assert.That(a, Is.EqualTo(b)); + Assert.That(a, Is.Not.EqualTo(c)); + } + + [Test] + public void DefaultBuffers_IsEmptyList () + { + var snapshot = new BufferIndexSnapshot + { + BufferCount = 0, + TotalLineCount = 0, + LruCacheCount = 0 + }; + + Assert.That(snapshot.Buffers, Is.Empty); + } +} \ No newline at end of file diff --git a/src/LogExpert.Tests/Buffers/BufferIndexTests.cs b/src/LogExpert.Tests/Buffers/BufferIndexTests.cs new file mode 100644 index 00000000..7df0ecfc --- /dev/null +++ b/src/LogExpert.Tests/Buffers/BufferIndexTests.cs @@ -0,0 +1,578 @@ +using ColumnizerLib; + +using LogExpert.Core.Classes.Log.Buffers; + +using Moq; + +using NUnit.Framework; + +namespace LogExpert.Tests.Buffers; + +/// +/// Unit tests for the extracted class. Uses in-memory LogBuffers with a fake ILogFileInfo — +/// no file I/O. +/// +[TestFixture] +internal class BufferIndexTests : IDisposable +{ + private const int MAX_BUFFERS = 50; + private const int LINES_PER_BUFFER = 500; + + private Mock _fakeFileInfo = null!; + private Mock _fakeFileInfo2 = null!; + private BufferIndex _index = null!; + + private bool _disposed; + + [SetUp] + public void SetUp () + { + _fakeFileInfo = new Mock(); + _ = _fakeFileInfo.Setup(f => f.FullName).Returns("fake1.log"); + _ = _fakeFileInfo.Setup(f => f.FileName).Returns("fake1.log"); + + _fakeFileInfo2 = new Mock(); + _ = _fakeFileInfo2.Setup(f => f.FullName).Returns("fake2.log"); + _ = _fakeFileInfo2.Setup(f => f.FileName).Returns("fake2.log"); + + _index = new BufferIndex(MAX_BUFFERS, LINES_PER_BUFFER); + } + + [TearDown] + public void TearDown () + { + _index.Dispose(); + } + + #region IDisposable Implementation + + public void Dispose () + { + Dispose(true); + GC.SuppressFinalize(this); + } + + protected virtual void Dispose (bool disposing) + { + if (_disposed) + { + return; + } + + if (disposing) + { + _index?.Dispose(); + } + + _disposed = true; + } + + #endregion + + #region Helper Methods + + /// + /// Creates a LogBuffer with the given startLine and lineCount, populated with dummy LogLines. + /// + private LogBuffer CreateBuffer (int startLine, int lineCount, ILogFileInfo? fileInfo = null) + { + var info = fileInfo ?? _fakeFileInfo.Object; + var buffer = new LogBuffer(info, LINES_PER_BUFFER) + { + StartLine = startLine, + StartPos = startLine * 100 + }; + + for (var j = 0; j < lineCount; j++) + { + buffer.AddLine(new LogLine($"line {startLine + j}".AsMemory(), startLine + j), (startLine + j) * 100); + } + + buffer.Size = lineCount * 100; + return buffer; + } + + /// + /// Populates the index with uniform buffers of lines. + /// + private void PopulateUniform (int count, int linesPerBuffer = LINES_PER_BUFFER, ILogFileInfo? fileInfo = null) + { + using var w = _index.AcquireWriteLock(); + for (var i = 0; i < count; i++) + { + _index.Add(CreateBuffer(i * linesPerBuffer, linesPerBuffer, fileInfo)); + } + } + + #endregion + + #region Lookup — 4-layer strategy + + [Test] + public void TryFindBuffer_EmptyIndex_ReturnsFalse () + { + using var r = _index.AcquireReadLock(); + Assert.That(_index.TryFindBuffer(0).Found, Is.False); + } + + [Test] + public void TryFindBuffer_SingleBuffer_FindsLine () + { + PopulateUniform(1, 10); + + using var r = _index.AcquireReadLock(); + Assert.That(_index.TryFindBuffer(0).Found, Is.True); + Assert.That(_index.TryFindBuffer(0).Buffer, Is.Not.Null); + Assert.That(_index.TryFindBuffer(0).Buffer!.StartLine, Is.EqualTo(0)); + Assert.That(_index.TryFindBuffer(0).Buffer!.LineCount, Is.EqualTo(10)); + } + + [Test] + public void TryFindBuffer_OutOfRange_ReturnsFalse () + { + PopulateUniform(2, 10); // lines 0–19 + + using var r = _index.AcquireReadLock(); + Assert.That(_index.TryFindBuffer(20).Found, Is.False); + Assert.That(_index.TryFindBuffer(-1).Found, Is.False); + } + + [TestCase(0, Description = "First line — Layer 0 or direct map")] + [TestCase(499, Description = "Last line of first buffer")] + [TestCase(500, Description = "First line of second buffer — boundary")] + [TestCase(4999, Description = "Last line overall")] + public void TryFindBuffer_VariousLines_FindsCorrectBuffer (int lineNum) + { + PopulateUniform(10); // 10 buffers × 500 lines = 5000 lines + + using var r = _index.AcquireReadLock(); + Assert.That(_index.TryFindBuffer(lineNum).Found, Is.True); + Assert.That(_index.TryFindBuffer(lineNum).Buffer, Is.Not.Null); + Assert.That(lineNum, Is.GreaterThanOrEqualTo(_index.TryFindBuffer(lineNum).Buffer!.StartLine)); + Assert.That(lineNum, Is.LessThan(_index.TryFindBuffer(lineNum).Buffer!.StartLine + _index.TryFindBuffer(lineNum).Buffer!.LineCount)); + } + + [Test] + public void TryFindBuffer_SequentialAccess_HitsThreadLocalCache () + { + PopulateUniform(5, 100); // 500 lines + + using var r = _index.AcquireReadLock(); + // First call sets thread-local cache + Assert.That(_index.TryFindBuffer(50).Found, Is.True); + // Second call within same buffer should hit Layer 0 + Assert.That(_index.TryFindBuffer(60).Found, Is.True); + Assert.That(_index.TryFindBuffer(60).Buffer!.StartLine, Is.EqualTo(0)); + } + + [Test] + public void TryFindBuffer_AdjacentForward_FindsNextBuffer () + { + PopulateUniform(3, 100); // lines 0–299 + + using var r = _index.AcquireReadLock(); + // Prime thread-local to buffer 0 + Assert.That(_index.TryFindBuffer(50).Found, Is.True); + // Cross boundary into buffer 1 — Layer 1 adjacent prediction + Assert.That(_index.TryFindBuffer(100).Found, Is.True); + Assert.That(_index.TryFindBuffer(100).Buffer!.StartLine, Is.EqualTo(100)); + } + + [Test] + public void TryFindBuffer_AdjacentBackward_FindsPrevBuffer () + { + PopulateUniform(3, 100); // lines 0–299 + + using var r = _index.AcquireReadLock(); + // Prime thread-local to buffer 1 + Assert.That(_index.TryFindBuffer(150).Found, Is.True); + // Cross backward into buffer 0 + Assert.That(_index.TryFindBuffer(50).Found, Is.True); + Assert.That(_index.TryFindBuffer(50).Buffer!.StartLine, Is.EqualTo(0)); + } + + [Test] + public void TryFindBuffer_RandomStride_FindsAllLines () + { + PopulateUniform(20, 100); // 2000 lines + + using var r = _index.AcquireReadLock(); + // Co-prime stride exercises Layers 2 and 3 + var stride = 701; + var lineNum = 0; + for (var i = 0; i < 2000; i++) + { + lineNum = (lineNum + stride) % 2000; + var logBufferEntry = _index.TryFindBuffer(lineNum); + Assert.That(logBufferEntry.Found, Is.True, + $"Failed to find buffer for line {lineNum}"); + Assert.That(lineNum, Is.GreaterThanOrEqualTo(logBufferEntry.Buffer!.StartLine)); + Assert.That(lineNum, Is.LessThan(logBufferEntry.Buffer.StartLine + logBufferEntry.Buffer.LineCount)); + } + } + + #endregion + + #region GetBufferForLineWithIndex + + [Test] + public void GetBufferForLineWithIndex_ReturnsBufferAndPositionalIndex () + { + PopulateUniform(3, 100); + + using var r = _index.AcquireReadLock(); + var logBufferEntry = _index.GetBufferForLineWithIndex(150); + Assert.That(logBufferEntry.Buffer, Is.Not.Null); + Assert.That(logBufferEntry.Buffer!.StartLine, Is.EqualTo(100)); + Assert.That(logBufferEntry.Index, Is.EqualTo(1)); + } + + [Test] + public void GetBufferForLineWithIndex_OutOfRange_ReturnsNull () + { + PopulateUniform(1, 10); + + using var r = _index.AcquireReadLock(); + var logBufferEntry = _index.GetBufferForLineWithIndex(999); + Assert.That(logBufferEntry.Buffer, Is.Null); + Assert.That(logBufferEntry.Index, Is.EqualTo(-1)); + } + + #endregion + + #region Mutation — Add / Remove / UpdateStartLine / Clear + + [Test] + public void Add_IncreasesBufferCount () + { + using var w = _index.AcquireWriteLock(); + Assert.That(_index.BufferCount, Is.EqualTo(0)); + + _index.Add(CreateBuffer(0, 10)); + Assert.That(_index.BufferCount, Is.EqualTo(1)); + + _index.Add(CreateBuffer(10, 10)); + Assert.That(_index.BufferCount, Is.EqualTo(2)); + } + + [Test] + public void Remove_DecreasesBufferCount () + { + var buf = CreateBuffer(0, 10); + using var w = _index.AcquireWriteLock(); + _index.Add(buf); + Assert.That(_index.BufferCount, Is.EqualTo(1)); + + var removed = _index.Remove(buf); + Assert.That(removed, Is.True); + Assert.That(_index.BufferCount, Is.EqualTo(0)); + } + + [Test] + public void Remove_NonExistentBuffer_ReturnsFalse () + { + PopulateUniform(1, 10); + var orphan = CreateBuffer(9999, 10); + + using var w = _index.AcquireWriteLock(); + Assert.That(_index.Remove(orphan), Is.False); + } + + [Test] + public void UpdateStartLine_MovesBuffer () + { + PopulateUniform(3, 100); // 0, 100, 200 + + using var w = _index.AcquireWriteLock(); + var buf = _index.GetBufferAt(1); // startLine=100 + Assert.That(buf.StartLine, Is.EqualTo(100)); + + _index.UpdateStartLine(buf, 50); + Assert.That(buf.StartLine, Is.EqualTo(50)); + + // Old key gone — no buffer starts at 100 anymore + Assert.That(_index.TryFindBuffer(100).Buffer?.StartLine, Is.Not.EqualTo(100)); + // New key present — buffer is reachable via its new start line + Assert.That(_index.TryFindBuffer(50).Found, Is.True); + Assert.That(_index.TryFindBuffer(50).Buffer, Is.SameAs(buf)); + } + + [Test] + public void Clear_RemovesAll () + { + PopulateUniform(5, 100); + + using var w = _index.AcquireWriteLock(); + Assert.That(_index.BufferCount, Is.EqualTo(5)); + + _index.Clear(); + Assert.That(_index.BufferCount, Is.EqualTo(0)); + Assert.That(_index.TotalLineCount, Is.EqualTo(0)); + Assert.That(_index.LruCacheCount, Is.EqualTo(0)); + } + + #endregion + + #region TotalLineCount - dirty/clean caching + + [Test] + public void TotalLineCount_ReflectsBufferContents () + { + PopulateUniform(3, 100); + + using var r = _index.AcquireReadLock(); + Assert.That(_index.TotalLineCount, Is.EqualTo(300)); + } + + [Test] + public void TotalLineCount_AfterMarkDirty_Recalculates () + { + PopulateUniform(2, 100); + + using var r = _index.AcquireReadLock(); + var first = _index.TotalLineCount; + Assert.That(first, Is.EqualTo(200)); + + // Mark dirty and re-read — should recalculate to same value + _index.MarkLineCountDirty(); + var second = _index.TotalLineCount; + Assert.That(second, Is.EqualTo(200)); + } + + #endregion + + #region Multi-file navigation + + [Test] + public void TryGetNextFileStartLine_TwoFiles_FindsBoundary () + { + // File1: buffers at 0, 100. File2: buffers at 200, 300. + using var w = _index.AcquireWriteLock(); + _index.Add(CreateBuffer(0, 100, _fakeFileInfo.Object)); + _index.Add(CreateBuffer(100, 100, _fakeFileInfo.Object)); + _index.Add(CreateBuffer(200, 100, _fakeFileInfo2.Object)); + _index.Add(CreateBuffer(300, 100, _fakeFileInfo2.Object)); + + Assert.That(_index.TryGetNextFileStartLine(50).Found, Is.True); + Assert.That(_index.TryGetNextFileStartLine(50).StartLine, Is.EqualTo(200)); + } + + [Test] + public void TryGetNextFileStartLine_LastFile_ReturnsFalse () + { + using var w = _index.AcquireWriteLock(); + _index.Add(CreateBuffer(0, 100, _fakeFileInfo.Object)); + _index.Add(CreateBuffer(100, 100, _fakeFileInfo2.Object)); + + // Line in second (last) file — no next file + Assert.That(_index.TryGetNextFileStartLine(150).Found, Is.False); + } + + [Test] + public void TryGetPrevFileStartLine_TwoFiles_FindsBoundary () + { + using var w = _index.AcquireWriteLock(); + _index.Add(CreateBuffer(0, 100, _fakeFileInfo.Object)); + _index.Add(CreateBuffer(100, 100, _fakeFileInfo.Object)); + _index.Add(CreateBuffer(200, 100, _fakeFileInfo2.Object)); + _index.Add(CreateBuffer(300, 100, _fakeFileInfo2.Object)); + + // Line 250 is in file2 — prev file ends at line 200 (startLine + lineCount of last file1 buffer) + Assert.That(_index.TryGetPrevFileStartLine(250).Found, Is.True); + Assert.That(_index.TryGetPrevFileStartLine(250).StartLine, Is.EqualTo(200)); + } + + [Test] + public void TryGetPrevFileStartLine_FirstFile_ReturnsFalse () + { + using var w = _index.AcquireWriteLock(); + _index.Add(CreateBuffer(0, 100, _fakeFileInfo.Object)); + _index.Add(CreateBuffer(100, 100, _fakeFileInfo2.Object)); + + Assert.That(_index.TryGetPrevFileStartLine(50).Found, Is.False); + } + + [Test] + public void GetFirstBufferForFile_ReturnsEarliestBuffer () + { + using var w = _index.AcquireWriteLock(); + _index.Add(CreateBuffer(0, 100, _fakeFileInfo.Object)); + _index.Add(CreateBuffer(100, 100, _fakeFileInfo.Object)); + _index.Add(CreateBuffer(200, 100, _fakeFileInfo2.Object)); + + var logBufferEntry = _index.GetBufferForLineWithIndex(150); + var first = _index.GetFirstBufferForFile(logBufferEntry.Buffer!, logBufferEntry.Index); + Assert.That(first!.StartLine, Is.EqualTo(0)); + } + + #endregion + + #region LRU Eviction + + [Test] + public void EvictLeastRecentlyUsed_BelowThreshold_DoesNothing () + { + PopulateUniform(5, 10); // well below MaxBuffers=50 + + using var r = _index.AcquireReadLock(); + // Touch all buffers + for (var i = 0; i < 50; i += 10) + { + _ = _index.TryFindBuffer(i); + } + + _index.EvictLeastRecentlyUsed(); + + // All buffers still have content + for (var i = 0; i < 5; i++) + { + Assert.That(_index.GetBufferAt(i).IsDisposed, Is.False); + } + } + + [Test] + public void EvictLeastRecentlyUsed_AboveThreshold_EvictsOldest () + { + // Use a small maxBuffers so we can exceed it easily + _index.Dispose(); + _index = new BufferIndex(maxBuffers: 3, maxLinesPerBuffer: 10); + + using var w = _index.AcquireWriteLock(); + // Add 20 buffers (way above maxBuffers=3) + for (var i = 0; i < 20; i++) + { + _index.Add(CreateBuffer(i * 10, 10)); + } + + // Touch only the last 3 buffers (make them "recent") + _ = _index.TryFindBuffer(170); + _ = _index.TryFindBuffer(180); + _ = _index.TryFindBuffer(190); + + _index.EvictLeastRecentlyUsed(); + + // LRU cache should be reduced toward maxBuffers + Assert.That(_index.LruCacheCount, Is.LessThanOrEqualTo(3 + 10)); + } + + #endregion + + #region ClearLru + + [Test] + public void ClearLru_ClearsIndexBeforeReturningToPool () + { + var pool = new LogBufferPool(100); + + using var w = _index.AcquireWriteLock(); + _index.Add(CreateBuffer(0, 10)); + _index.Add(CreateBuffer(10, 10)); + + // Touch buffers to populate LRU + _ = _index.TryFindBuffer(0); + _ = _index.TryFindBuffer(10); + + Assert.That(_index.BufferCount, Is.EqualTo(2)); + Assert.That(_index.LruCacheCount, Is.GreaterThan(0)); + + _index.ClearLru(pool); + + // Index is empty — this is the bug fix: index clears FIRST + Assert.That(_index.BufferCount, Is.EqualTo(0)); + Assert.That(_index.LruCacheCount, Is.EqualTo(0)); + Assert.That(_index.TotalLineCount, Is.EqualTo(0)); + + // Verify no lookup succeeds after ClearLru (prevents stale reference bug) + Assert.That(_index.TryFindBuffer(0).Found, Is.False); + Assert.That(_index.TryFindBuffer(10).Found, Is.False); + } + + [Test] + public void ClearLru_BuffersReturnedToPool () + { + var pool = new LogBufferPool(100); + + using var w = _index.AcquireWriteLock(); + var buf1 = CreateBuffer(0, 10); + var buf2 = CreateBuffer(10, 10); + _index.Add(buf1); + _index.Add(buf2); + + // Touch to populate LRU + _ = _index.TryFindBuffer(0); + _ = _index.TryFindBuffer(10); + + _index.ClearLru(pool); + + // Buffers should be disposed (returned to pool disposes content) + Assert.That(buf1.IsDisposed, Is.True); + Assert.That(buf2.IsDisposed, Is.True); + } + + #endregion + + #region SnapShot + + [Test] + public void CreateSnapshot_CapturesCurrentState () + { + PopulateUniform(3, 100); + + var snapshot = _index.CreateSnapshot(); + Assert.That(snapshot.BufferCount, Is.EqualTo(3)); + Assert.That(snapshot.TotalLineCount, Is.EqualTo(300)); + Assert.That(snapshot.Buffers, Has.Count.EqualTo(3)); + + Assert.That(snapshot.Buffers[0].StartLine, Is.EqualTo(0)); + Assert.That(snapshot.Buffers[0].LineCount, Is.EqualTo(100)); + Assert.That(snapshot.Buffers[0].FileName, Is.EqualTo("fake1.log")); + + Assert.That(snapshot.Buffers[1].StartLine, Is.EqualTo(100)); + Assert.That(snapshot.Buffers[2].StartLine, Is.EqualTo(200)); + } + + [Test] + public void CreateSnapshot_ImmutableAfterModification () + { + PopulateUniform(2, 100); + + var snapshot = _index.CreateSnapshot(); + Assert.That(snapshot.BufferCount, Is.EqualTo(2)); + + // Add more buffers after snapshot + using var w = _index.AcquireWriteLock(); + _index.Add(CreateBuffer(200, 100)); + + // Snapshot is unchanged + Assert.That(snapshot.BufferCount, Is.EqualTo(2)); + Assert.That(snapshot.Buffers, Has.Count.EqualTo(2)); + } + + #endregion + + #region LockScopeTests + + [Test] + public void UpgradeableReadLock_CanUpgradeToWrite () + { + using var upgradeable = _index.AcquireUpgradeableReadLock(); + _index.Add(CreateBuffer(0, 10)); // Add requires at least upgradeable + + using (upgradeable.UpgradeToWrite()) + { + _index.Add(CreateBuffer(10, 10)); + } + + // Back to upgradeable-read + Assert.That(_index.BufferCount, Is.EqualTo(2)); + } + + [Test] + public void Dispose_CanBeCalledMultipleTimes () + { + _index.Dispose(); + Assert.DoesNotThrow(_index.Dispose); + } + + #endregion +} \ No newline at end of file diff --git a/src/LogExpert.Tests/BufferShiftTest.cs b/src/LogExpert.Tests/Buffers/BufferShiftTest.cs similarity index 84% rename from src/LogExpert.Tests/BufferShiftTest.cs rename to src/LogExpert.Tests/Buffers/BufferShiftTest.cs index b98367ec..8233da59 100644 --- a/src/LogExpert.Tests/BufferShiftTest.cs +++ b/src/LogExpert.Tests/Buffers/BufferShiftTest.cs @@ -7,7 +7,7 @@ using NUnit.Framework; -namespace LogExpert.Tests; +namespace LogExpert.Tests.Buffers; [TestFixture] internal class BufferShiftTest : RolloverHandlerTestBase @@ -94,12 +94,12 @@ public void TestShiftBuffers1 (ReaderType readerType) enumerator = files.GetEnumerator(); _ = enumerator.MoveNext(); - var logBuffers = reader.GetBufferList(); + var snapshot = reader.BufferIndex.CreateSnapshot(); var startLine = 0; - foreach (var logBuffer in logBuffers) + foreach (var logBuffer in snapshot.Buffers) { - Assert.That(enumerator.Current, Is.EqualTo(logBuffer.FileInfo.FullName)); + Assert.That(enumerator.Current, Is.EqualTo(logBuffer.FileName)); Assert.That(logBuffer.StartLine, Is.EqualTo(startLine)); startLine += 10; _ = enumerator.MoveNext(); @@ -109,36 +109,37 @@ public void TestShiftBuffers1 (ReaderType readerType) enumerator = files.GetEnumerator(); _ = enumerator.MoveNext(); _ = enumerator.MoveNext(); // move to 2nd entry. The first file now contains 2nd file's content (because rollover) - logBuffers = reader.GetBufferList(); + + snapshot = reader.BufferIndex.CreateSnapshot(); int i; - for (i = 0; i < logBuffers.Count - 2; ++i) + for (i = 0; i < snapshot.Buffers.Count - 2; ++i) { - var logBuffer = logBuffers[i]; - var line = logBuffer.GetLineMemoryOfBlock(0); - if (!line.HasValue) + var logBuffer = snapshot.Buffers[i]; + var line = reader.GetLogLineMemory(logBuffer.StartLine); + if (line == null) { Assert.Fail("Expected first block line to be present."); continue; } - Assert.That(line.Value.FullLine.Span.Contains(enumerator.Current.AsSpan(), StringComparison.Ordinal)); + Assert.That(line.FullLine.Span.Contains(enumerator.Current.AsSpan(), StringComparison.Ordinal)); _ = enumerator.MoveNext(); } // the last 2 files now contain the content of the previously watched file - for (; i < logBuffers.Count; ++i) + for (; i < snapshot.Buffers.Count; ++i) { - var logBuffer = logBuffers[i]; - var line = logBuffer.GetLineMemoryOfBlock(0); + var logBuffer = snapshot.Buffers[i]; + var line = reader.GetLogLineMemory(logBuffer.StartLine); - if (!line.HasValue) + if (line == null) { Assert.Fail("Expected first block line to be present."); continue; } - Assert.That(line.Value.FullLine.Span.Contains(enumerator.Current.AsSpan(), StringComparison.Ordinal)); + Assert.That(line.FullLine.Span.Contains(enumerator.Current.AsSpan(), StringComparison.Ordinal)); } oldCount = lil.Count; diff --git a/src/LogExpert.Tests/Buffers/CharBlockAllocatorTests.cs b/src/LogExpert.Tests/Buffers/CharBlockAllocatorTests.cs new file mode 100644 index 00000000..b0004884 --- /dev/null +++ b/src/LogExpert.Tests/Buffers/CharBlockAllocatorTests.cs @@ -0,0 +1,130 @@ +using System.Buffers; + +using LogExpert.Core.Classes.Log.Buffers; + +using NUnit.Framework; + +namespace LogExpert.Tests.Buffers; + +[TestFixture] +public class CharBlockAllocatorTests +{ + [Test] + public void Rent_SmallAllocation_ReturnsMemoryFromSameBlock () + { + using var allocator = new CharBlockAllocator(1024); + + var mem1 = allocator.Rent(100); + var mem2 = allocator.Rent(100); + + Assert.That(allocator.BlockCount, Is.EqualTo(1)); + Assert.That(mem1.Length, Is.EqualTo(100)); + Assert.That(mem2.Length, Is.EqualTo(100)); + } + + [Test] + public void Rent_ExceedsBlock_AllocatesNewBlock () + { + using var allocator = new CharBlockAllocator(128); + + var mem1 = allocator.Rent(100); + var mem2 = allocator.Rent(100); // won't fit in first block + + Assert.That(allocator.BlockCount, Is.EqualTo(2)); + Assert.That(mem1.Length, Is.EqualTo(100)); + Assert.That(mem2.Length, Is.EqualTo(100)); + } + + [Test] + public void Rent_OversizedLine_GetsStandaloneArray () + { + using var allocator = new CharBlockAllocator(128); + + var mem = allocator.Rent(256); + + Assert.That(allocator.BlockCount, Is.EqualTo(1)); // normal blocks unchanged + Assert.That(allocator.OversizedBlockCount, Is.EqualTo(1)); // tracked separately + Assert.That(mem.Length, Is.EqualTo(256)); + } + + [Test] + public void Rent_ZeroLength_ReturnsEmpty () + { + using var allocator = new CharBlockAllocator(128); + + var mem = allocator.Rent(0); + + Assert.That(mem.IsEmpty, Is.True); + } + + [Test] + public void DetachBlocks_ReturnsNormalBlocks_ResetsAllocator () + { + using var allocator = new CharBlockAllocator(128); + + _ = allocator.Rent(100); + _ = allocator.Rent(100); // triggers second block + + var blocks = allocator.DetachBlocks(); + + Assert.That(blocks, Has.Count.EqualTo(2)); + Assert.That(allocator.BlockCount, Is.EqualTo(1)); // fresh block created + } + + [Test] + public void DetachBlocks_ReturnsOversizedImmediately () + { + using var allocator = new CharBlockAllocator(128); + + _ = allocator.Rent(100); // normal + _ = allocator.Rent(256); // oversized + _ = allocator.Rent(50); // normal (fits in second block) + + Assert.That(allocator.OversizedBlockCount, Is.EqualTo(1)); + + var blocks = allocator.DetachBlocks(); + + // All blocks (normal + oversized) are transferred to caller + Assert.That(blocks, Has.Count.EqualTo(3)); // initial + second normal + oversized + Assert.That(allocator.OversizedBlockCount, Is.EqualTo(0)); // transferred to caller + } + + [Test] + public void ReturnAll_ReturnsBlocksToPool () + { + var allocator = new CharBlockAllocator(128); + + _ = allocator.Rent(100); + _ = allocator.Rent(100); + + allocator.ReturnAll(); + + Assert.That(allocator.BlockCount, Is.EqualTo(0)); + } + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Tests")] + public void SlicesAreIndependent () + { + using var allocator = new CharBlockAllocator(1024); + + var mem1 = allocator.Rent(5); + "Hello".AsSpan().CopyTo(mem1.Span); + + var mem2 = allocator.Rent(5); + "World".AsSpan().CopyTo(mem2.Span); + + Assert.That(mem1.Span.ToString(), Is.EqualTo("Hello")); + Assert.That(mem2.Span.ToString(), Is.EqualTo("World")); + } + + [Test] + public void Dispose_IsIdempotent () + { + var allocator = new CharBlockAllocator(128); + _ = allocator.Rent(64); + + allocator.Dispose(); + allocator.Dispose(); // should not throw + } +} \ No newline at end of file diff --git a/src/LogExpert.Tests/Buffers/LogBufferCharBlockTests.cs b/src/LogExpert.Tests/Buffers/LogBufferCharBlockTests.cs new file mode 100644 index 00000000..15072de0 --- /dev/null +++ b/src/LogExpert.Tests/Buffers/LogBufferCharBlockTests.cs @@ -0,0 +1,228 @@ +using System.Buffers; + +using ColumnizerLib; + +using LogExpert.Core.Classes.Log.Buffers; + +using Moq; + +using NUnit.Framework; + +namespace LogExpert.Tests.Buffers; + +[TestFixture] +public class LogBufferCharBlockTests +{ + private Mock _mockFileInfo; + + [SetUp] + public void Setup () + { + _mockFileInfo = new Mock(); + _ = _mockFileInfo.Setup(f => f.FullName).Returns("test.log"); + } + + [Test] + public void AttachCharBlocks_AcceptsBlockList () + { + var buffer = new LogBuffer(_mockFileInfo.Object, 10); + var blocks = new List + { + ArrayPool.Shared.Rent(128), + ArrayPool.Shared.Rent(128) + }; + + // Should not throw + buffer.AttachCharBlocks(blocks); + + // Clean up via eviction + buffer.EvictContent(); + } + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Tests")] + public void EvictContent_WhilePinned_TriggersDebugAssert () + { + var buffer = new LogBuffer(_mockFileInfo.Object, 10); + var block = ArrayPool.Shared.Rent(128); + buffer.AttachCharBlocks([block]); + buffer.AddLine(new LogLine("test".AsMemory(), 0), 0); + buffer.Pin(); + + // In Debug builds, this should trigger the assert. + // In Release builds, it proceeds (defense in depth via eviction skip). + // We can't directly test Debug.Assert in NUnit, but we verify the behavior: + // After eviction while pinned, the buffer should still be disposed + // (the assert is a developer warning, not a runtime guard). + buffer.EvictContent(); + buffer.Unpin(); + } + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Tests")] + public void EvictContent_ReturnsAttachedCharBlocks () + { + var buffer = new LogBuffer(_mockFileInfo.Object, 10); + var block = ArrayPool.Shared.Rent(128); + buffer.AttachCharBlocks([block]); + + // Add a line so there's something to evict + var lineMemory = "test line".AsMemory(); + buffer.AddLine(new LogLine(lineMemory, 0), 0); + + buffer.EvictContent(); + + // After eviction, the block should have been returned to the pool. + // We can't directly assert pool return, but we can verify no exception + // and that re-attaching new blocks works. + var newBlock = ArrayPool.Shared.Rent(128); + buffer.AttachCharBlocks([newBlock]); + buffer.EvictContent(); // clean up + } + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Tests")] + public void DisposeContent_ReturnsAttachedCharBlocks () + { + var buffer = new LogBuffer(_mockFileInfo.Object, 10); + var block = ArrayPool.Shared.Rent(128); + buffer.AttachCharBlocks([block]); + + buffer.AddLine(new LogLine("test".AsMemory(), 0), 0); + + buffer.DisposeContent(); + + // Should be able to reinitialise and attach new blocks + buffer.Reinitialise(_mockFileInfo.Object, 10); + var newBlock = ArrayPool.Shared.Rent(128); + buffer.AttachCharBlocks([newBlock]); + buffer.EvictContent(); + } + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Tests")] + public void ClearLines_ReturnsAttachedCharBlocks () + { + var buffer = new LogBuffer(_mockFileInfo.Object, 10); + var block = ArrayPool.Shared.Rent(128); + buffer.AttachCharBlocks([block]); + + buffer.AddLine(new LogLine("test".AsMemory(), 0), 0); + + buffer.ClearLines(); + + Assert.That(buffer.LineCount, Is.EqualTo(0)); + + // Attach new blocks after clear + var newBlock = ArrayPool.Shared.Rent(128); + buffer.AttachCharBlocks([newBlock]); + buffer.EvictContent(); + } + + [Test] + public void Reinitialise_ReturnsAttachedCharBlocks () + { + var buffer = new LogBuffer(_mockFileInfo.Object, 10); + var block = ArrayPool.Shared.Rent(128); + buffer.AttachCharBlocks([block]); + + buffer.Reinitialise(_mockFileInfo.Object, 10); + + // After reinitialise, old blocks should be returned. + // Verify by attaching new blocks (no double-return crash). + var newBlock = ArrayPool.Shared.Rent(128); + buffer.AttachCharBlocks([newBlock]); + buffer.EvictContent(); + } + + [Test] + public void AttachCharBlocks_ReturnsOldBlocks_WhenCalledTwice () + { + var buffer = new LogBuffer(_mockFileInfo.Object, 10); + + var block1 = ArrayPool.Shared.Rent(128); + buffer.AttachCharBlocks([block1]); + + // Attaching new blocks should return the old ones first + var block2 = ArrayPool.Shared.Rent(128); + buffer.AttachCharBlocks([block2]); + + buffer.EvictContent(); + } + + [Test] + public void AttachCharBlocks_Null_DoesNotThrow () + { + var buffer = new LogBuffer(_mockFileInfo.Object, 10); + + buffer.AttachCharBlocks(null); + buffer.EvictContent(); // should not throw + } + + [Test] + public void AttachCharBlocks_EmptyList_DoesNotThrow () + { + var buffer = new LogBuffer(_mockFileInfo.Object, 10); + + buffer.AttachCharBlocks([]); + buffer.EvictContent(); // should not throw + } + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Tests")] + public void EvictContent_WithoutAttachedBlocks_DoesNotThrow () + { + var buffer = new LogBuffer(_mockFileInfo.Object, 10); + buffer.AddLine(new LogLine("test".AsMemory(), 0), 0); + + // No char blocks attached — should evict cleanly (backward compat) + buffer.EvictContent(); + } + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Tests")] + public void BlockBackedLines_ContentSurvivesUntilEviction () + { + var buffer = new LogBuffer(_mockFileInfo.Object, 10); + + // Simulate block-based allocation: rent a block, write lines into it + var block = ArrayPool.Shared.Rent(1024); + "Hello World".AsSpan().CopyTo(block.AsSpan(0, 11)); + "Second Line".AsSpan().CopyTo(block.AsSpan(11, 11)); + + var line1Memory = new ReadOnlyMemory(block, 0, 11); + var line2Memory = new ReadOnlyMemory(block, 11, 11); + + buffer.AddLine(new LogLine(line1Memory, 0), 0); + buffer.AddLine(new LogLine(line2Memory, 1), 11); + buffer.AttachCharBlocks([block]); + + // Lines should be readable while buffer is alive + var retrieved1 = buffer.GetLineMemoryOfBlock(0); + var retrieved2 = buffer.GetLineMemoryOfBlock(1); + Assert.That(retrieved1.HasValue, Is.True); + Assert.That(retrieved2.HasValue, Is.True); + + if (retrieved1 is not { } retrievedLine1) + { + Assert.Fail("Expected first retrieved line to have a value."); + } + else + { + Assert.That(retrievedLine1.FullLine.Span.ToString(), Is.EqualTo("Hello World")); + } + + if (retrieved2 is not { } retrievedLine2) + { + Assert.Fail("Expected second retrieved line to have a value."); + } + else + { + Assert.That(retrievedLine2.FullLine.Span.ToString(), Is.EqualTo("Second Line")); + } + + // After eviction, blocks are returned and lines are no longer accessible + buffer.EvictContent(); + Assert.That(buffer.IsDisposed, Is.True); + } +} \ No newline at end of file diff --git a/src/LogExpert.Tests/Buffers/LogBufferPinningTests.cs b/src/LogExpert.Tests/Buffers/LogBufferPinningTests.cs new file mode 100644 index 00000000..afc180d2 --- /dev/null +++ b/src/LogExpert.Tests/Buffers/LogBufferPinningTests.cs @@ -0,0 +1,193 @@ +using ColumnizerLib; + +using LogExpert.Core.Classes.Log.Buffers; + +using Moq; + +using NUnit.Framework; + +namespace LogExpert.Tests.Buffers; + +[TestFixture] +internal class LogBufferPinningTests +{ + private Mock _fakeFileInfo = null!; + + [SetUp] + public void SetUp () + { + _fakeFileInfo = new Mock(); + _ = _fakeFileInfo.Setup(f => f.FullName).Returns("fake.log"); + _ = _fakeFileInfo.Setup(f => f.FileName).Returns("fake.log"); + } + + #region LogBuffer Pin/Unpin + + [Test] + public void NewBuffer_IsNotPinned () + { + var buffer = new LogBuffer(_fakeFileInfo.Object, 10); + Assert.That(buffer.IsPinned, Is.False); + } + + [Test] + public void Pin_MakesBufferPinned () + { + var buffer = new LogBuffer(_fakeFileInfo.Object, 10); + buffer.Pin(); + Assert.That(buffer.IsPinned, Is.True); + buffer.Unpin(); + } + + [Test] + public void PinUnpin_ReturnsToUnpinned () + { + var buffer = new LogBuffer(_fakeFileInfo.Object, 10); + buffer.Pin(); + buffer.Unpin(); + Assert.That(buffer.IsPinned, Is.False); + } + + [Test] + public void MultiplePin_RequiresMatchingUnpins () + { + var buffer = new LogBuffer(_fakeFileInfo.Object, 10); + buffer.Pin(); + buffer.Pin(); + buffer.Unpin(); + Assert.That(buffer.IsPinned, Is.True, "Still pinned after one unpin of two pins"); + buffer.Unpin(); + Assert.That(buffer.IsPinned, Is.False, "Unpinned after matching unpins"); + } + + [Test] + public void Reinitialise_ResetsPinCount () + { + var buffer = new LogBuffer(_fakeFileInfo.Object, 10); + buffer.Pin(); + buffer.Pin(); + buffer.Reinitialise(_fakeFileInfo.Object, 10); + Assert.That(buffer.IsPinned, Is.False); + } + + #endregion + + #region BufferIndex eviction with pins + + [Test] + public void EvictLeastRecentlyUsed_SkipsPinnedBuffer () + { + using var index = new BufferIndex(maxBuffers: 3, maxLinesPerBuffer: 10); + + using var w = index.AcquireWriteLock(); + + // Add 20 buffers (way above maxBuffers=3) + for (var i = 0; i < 20; i++) + { + var buf = new LogBuffer(_fakeFileInfo.Object, 10) { StartLine = i * 10, StartPos = i * 100 }; + for (var j = 0; j < 10; j++) + { + buf.AddLine(new LogLine($"line {i * 10 + j}".AsMemory(), i * 10 + j), (i * 10 + j) * 10); + } + buf.Size = 100; + index.Add(buf); + } + + // Pin the first buffer (oldest = first to be evicted) + var firstEntry = index.TryFindBuffer(0); + Assert.That(firstEntry.Found, Is.True); + firstEntry.Buffer!.Pin(); + + index.EvictLeastRecentlyUsed(); + + // The pinned buffer should NOT be evicted + Assert.That(firstEntry.Buffer.IsDisposed, Is.False, "Pinned buffer should survive eviction"); + Assert.That(firstEntry.Buffer.IsPinned, Is.True); + + // Clean up + firstEntry.Buffer.Unpin(); + } + + [Test] + public void EvictLeastRecentlyUsed_EvictsAfterUnpin () + { + using var index = new BufferIndex(maxBuffers: 3, maxLinesPerBuffer: 10); + + using var w = index.AcquireWriteLock(); + + // Add 20 buffers (way above maxBuffers=3) + for (var i = 0; i < 20; i++) + { + var buf = new LogBuffer(_fakeFileInfo.Object, 10) { StartLine = i * 10, StartPos = i * 100 }; + for (var j = 0; j < 10; j++) + { + buf.AddLine(new LogLine($"line {i * 10 + j}".AsMemory(), i * 10 + j), (i * 10 + j) * 10); + } + buf.Size = 100; + index.Add(buf); + } + + // Pin buffer 0, evict (it survives), then unpin + var firstEntry = index.TryFindBuffer(0); + firstEntry.Buffer!.Pin(); + index.EvictLeastRecentlyUsed(); + Assert.That(firstEntry.Buffer.IsDisposed, Is.False); + firstEntry.Buffer.Unpin(); + + // After first eviction, LRU count is near maxBuffers (~3-4 entries). + // EvictLeastRecentlyUsed has a threshold guard: only fires when count > maxBuffers + 10. + // Add fresh buffers to push LRU count above that threshold. + for (var i = 20; i < 40; i++) + { + var buf = new LogBuffer(_fakeFileInfo.Object, 10) { StartLine = i * 10, StartPos = i * 100 }; + for (var j = 0; j < 10; j++) + { + buf.AddLine(new LogLine($"line {i * 10 + j}".AsMemory(), i * 10 + j), (i * 10 + j) * 10); + } + buf.Size = 100; + index.Add(buf); + } + + // Touch new buffers to populate LRU + for (var i = 20; i < 40; i++) + { + _ = index.TryFindBuffer(i * 10); + } + + index.EvictLeastRecentlyUsed(); + + Assert.That(firstEntry.Buffer.IsDisposed, Is.True, "Unpinned buffer should be evicted"); + } + + [Test] + public void EvictLeastRecentlyUsed_UnpinnedBuffersStillEvicted () + { + using var index = new BufferIndex(maxBuffers: 3, maxLinesPerBuffer: 10); + + using var w = index.AcquireWriteLock(); + + for (var i = 0; i < 20; i++) + { + var buf = new LogBuffer(_fakeFileInfo.Object, 10) { StartLine = i * 10, StartPos = i * 100 }; + for (var j = 0; j < 10; j++) + { + buf.AddLine(new LogLine($"line {i * 10 + j}".AsMemory(), i * 10 + j), (i * 10 + j) * 10); + } + + buf.Size = 100; + index.Add(buf); + } + + // Touch only the last 3 to make them recent + _ = index.TryFindBuffer(170); + _ = index.TryFindBuffer(180); + _ = index.TryFindBuffer(190); + + index.EvictLeastRecentlyUsed(); + + // LRU should have reduced — unpinned old buffers evicted + Assert.That(index.LruCacheCount, Is.LessThanOrEqualTo(3 + 10)); + } + + #endregion +} \ No newline at end of file diff --git a/src/LogExpert.Tests/Buffers/PinHandleTests.cs b/src/LogExpert.Tests/Buffers/PinHandleTests.cs new file mode 100644 index 00000000..e63d576c --- /dev/null +++ b/src/LogExpert.Tests/Buffers/PinHandleTests.cs @@ -0,0 +1,206 @@ +using ColumnizerLib; + +using LogExpert.Core.Classes.Log.Buffers; + +using Moq; + +using NUnit.Framework; + +namespace LogExpert.Tests.Buffers; + +[TestFixture] +internal class PinHandleTests +{ + private Mock _fakeFileInfo = null!; + + [SetUp] + public void SetUp () + { + _fakeFileInfo = new Mock(); + _ = _fakeFileInfo.Setup(f => f.FullName).Returns("fake.log"); + _ = _fakeFileInfo.Setup(f => f.FileName).Returns("fake.log"); + } + + private LogBuffer CreateBuffer (int startLine, int lineCount) + { + var buffer = new LogBuffer(_fakeFileInfo.Object, 500) + { + StartLine = startLine, + StartPos = startLine * 100 + }; + + for (var j = 0; j < lineCount; j++) + { + buffer.AddLine(new LogLine($"line {startLine + j}".AsMemory(), startLine + j), (startLine + j) * 100); + } + + buffer.Size = lineCount * 100; + return buffer; + } + + #region PinHandle + + [Test] + public void Dispose_UnpinsAllBuffers () + { + var buf1 = CreateBuffer(0, 10); + var buf2 = CreateBuffer(10, 10); + buf1.Pin(); + buf2.Pin(); + + // Note: PinHandle constructor is internal, test accesses via InternalsVisibleTo or PinRange + var handle = new PinHandle([buf1, buf2]); + + handle.Dispose(); + + Assert.That(buf1.IsPinned, Is.False); + Assert.That(buf2.IsPinned, Is.False); + } + + [Test] + public void DoubleDispose_IsNoop () + { + var buf = CreateBuffer(0, 10); + buf.Pin(); + + var handle = new PinHandle([buf]); + handle.Dispose(); + handle.Dispose(); // should not throw or double-unpin + + Assert.That(buf.IsPinned, Is.False); + } + + [Test] + public void EmptyHandle_DisposeIsNoop () + { + var handle = new PinHandle([]); + handle.Dispose(); // should not throw + } + + #endregion + + #region BufferIndex.PinRange + + [Test] + public void PinRange_SingleBuffer_PinsIt () + { + using var index = new BufferIndex(50, 500); + using var w = index.AcquireWriteLock(); + + index.Add(CreateBuffer(0, 10)); + + using var handle = index.PinRange(0, 9); + + Assert.That(handle.Count, Is.EqualTo(1)); + var entry = index.TryFindBuffer(0); + Assert.That(entry.Buffer!.IsPinned, Is.True); + } + + [Test] + public void PinRange_MultipleBuffers_PinsAll () + { + using var index = new BufferIndex(50, 500); + using var w = index.AcquireWriteLock(); + + index.Add(CreateBuffer(0, 10)); + index.Add(CreateBuffer(10, 10)); + index.Add(CreateBuffer(20, 10)); + + using var handle = index.PinRange(0, 29); + + Assert.That(handle.Count, Is.EqualTo(3)); + } + + [Test] + public void PinRange_EmptyIndex_ReturnsEmptyHandle () + { + using var index = new BufferIndex(50, 500); + using var w = index.AcquireWriteLock(); + + using var handle = index.PinRange(0, 10); + + Assert.That(handle.Count, Is.EqualTo(0)); + } + + [Test] + public void PinRange_PartialOverlap_PinsOnlyExisting () + { + using var index = new BufferIndex(50, 500); + using var w = index.AcquireWriteLock(); + + index.Add(CreateBuffer(0, 10)); + + // Range extends past available buffers + using var handle = index.PinRange(0, 100); + + Assert.That(handle.Count, Is.EqualTo(1)); + } + + [Test] + public void DisposeHandle_UnpinsBuffers_AllowsEviction () + { + using var index = new BufferIndex(maxBuffers: 3, maxLinesPerBuffer: 10); + using var w = index.AcquireWriteLock(); + + for (var i = 0; i < 20; i++) + { + index.Add(CreateBuffer(i * 10, 10)); + } + + // Pin first buffer + var handle = index.PinRange(0, 9); + index.EvictLeastRecentlyUsed(); + + var entry = index.TryFindBuffer(0); + Assert.That(entry.Buffer!.IsDisposed, Is.False, "Pinned buffer should survive"); + + // Dispose handle (unpin), then re-populate LRU above eviction threshold + handle.Dispose(); + + // After first eviction, LRU count is ~3-4. EvictLeastRecentlyUsed requires + // count > maxBuffers + 10 = 13 to trigger. Add fresh buffers to exceed that. + for (var i = 20; i < 40; i++) + { + index.Add(CreateBuffer(i * 10, 10)); + } + + // Touch new buffers to populate LRU + for (var i = 20; i < 40; i++) + { + _ = index.TryFindBuffer(i * 10); + } + + index.EvictLeastRecentlyUsed(); + + Assert.That(entry.Buffer.IsDisposed, Is.True, "Unpinned buffer should be evicted"); + } + + [Test] + public void SequentialPrefetchPattern_OldPinsReleased () + { + using var index = new BufferIndex(50, 500); + using var w = index.AcquireWriteLock(); + + index.Add(CreateBuffer(0, 10)); + index.Add(CreateBuffer(10, 10)); + index.Add(CreateBuffer(20, 10)); + + // Simulate first prefetch: pin buffer 0 + var handle1 = index.PinRange(0, 9); + var buf0 = index.TryFindBuffer(0).Buffer!; + Assert.That(buf0.IsPinned, Is.True); + + // Simulate scroll: dispose old, pin new range + handle1.Dispose(); + Assert.That(buf0.IsPinned, Is.False); + + var handle2 = index.PinRange(10, 19); + var buf1 = index.TryFindBuffer(10).Buffer!; + Assert.That(buf1.IsPinned, Is.True); + Assert.That(buf0.IsPinned, Is.False); + + handle2.Dispose(); + } + + #endregion +} \ No newline at end of file diff --git a/src/LogExpert.Tests/CSVColumnizerTest.cs b/src/LogExpert.Tests/ColumnizerTests/CSVColumnizerTest.cs similarity index 98% rename from src/LogExpert.Tests/CSVColumnizerTest.cs rename to src/LogExpert.Tests/ColumnizerTests/CSVColumnizerTest.cs index 24fae246..35445f9b 100644 --- a/src/LogExpert.Tests/CSVColumnizerTest.cs +++ b/src/LogExpert.Tests/ColumnizerTests/CSVColumnizerTest.cs @@ -8,7 +8,7 @@ using NUnit.Framework; -namespace LogExpert.Tests; +namespace LogExpert.Tests.ColumnizerTests; [TestFixture] public class CSVColumnizerTest diff --git a/src/LogExpert.Tests/ColumnizerJsonConverterTests.cs b/src/LogExpert.Tests/ColumnizerTests/ColumnizerJsonConverterTests.cs similarity index 95% rename from src/LogExpert.Tests/ColumnizerJsonConverterTests.cs rename to src/LogExpert.Tests/ColumnizerTests/ColumnizerJsonConverterTests.cs index 4645bb7e..aae21480 100644 --- a/src/LogExpert.Tests/ColumnizerJsonConverterTests.cs +++ b/src/LogExpert.Tests/ColumnizerTests/ColumnizerJsonConverterTests.cs @@ -7,7 +7,7 @@ using NUnit.Framework; -namespace LogExpert.Tests; +namespace LogExpert.Tests.ColumnizerTests; public class MockColumnizer : ILogLineMemoryColumnizer { @@ -54,6 +54,8 @@ public void SetConfig (object config) { } public DateTime GetTimestamp (ILogLineMemoryColumnizerCallback callback, ILogLineMemory logLine) => throw new NotImplementedException(); public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, string oldValue) => throw new NotImplementedException(); + + public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, ReadOnlyMemory oldValue) => throw new NotImplementedException(); } public class MockColumnizerWithCustomName : ILogLineMemoryColumnizer @@ -95,6 +97,12 @@ public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, st { throw new NotImplementedException(); } + + public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, ReadOnlyMemory oldValue) + { + throw new NotImplementedException(); + } + } [TestFixture] diff --git a/src/LogExpert.Tests/ColumnizerPickerTest.cs b/src/LogExpert.Tests/ColumnizerTests/ColumnizerPickerTest.cs similarity index 97% rename from src/LogExpert.Tests/ColumnizerPickerTest.cs rename to src/LogExpert.Tests/ColumnizerTests/ColumnizerPickerTest.cs index e6ff80c3..3d69741e 100644 --- a/src/LogExpert.Tests/ColumnizerPickerTest.cs +++ b/src/LogExpert.Tests/ColumnizerTests/ColumnizerPickerTest.cs @@ -1,162 +1,162 @@ -using System.Reflection; - -using ColumnizerLib; - -using LogExpert.Core.Classes.Columnizer; -using LogExpert.Core.Classes.Log; -using LogExpert.Core.Entities; -using LogExpert.Core.Enums; - -using Moq; - -using NUnit.Framework; - -namespace LogExpert.Tests; - -/// -/// Summary description for AutoColumnizerTest -/// -[TestFixture] -public class ColumnizerPickerTest -{ - [SetUp] - public void Setup () - { - // Reset singleton for testing (same pattern as PluginRegistryTests) - ResetPluginRegistrySingleton(); - - // Initialize plugin registry with proper test directory - var testDataPath = Path.Join(Path.GetTempPath(), "LogExpertTests", Guid.NewGuid().ToString()); - _ = Directory.CreateDirectory(testDataPath); - - var pluginRegistry = PluginRegistry.PluginRegistry.Create(testDataPath, 250); - - // Verify the local file system plugin is registered - var localPlugin = pluginRegistry.FindFileSystemForUri(@"C:\test.txt"); - Assert.That(localPlugin, Is.Not.Null, "Local file system plugin not registered!"); - } - - [TearDown] - public void TearDown () - { - ResetPluginRegistrySingleton(); - } - - /// - /// Uses reflection to reset the singleton instance for testing. - /// This ensures each test starts with a fresh PluginRegistry state. - /// - private static void ResetPluginRegistrySingleton () - { - var instanceField = typeof(PluginRegistry.PluginRegistry).GetField("_instance", BindingFlags.Static | BindingFlags.NonPublic); - instanceField?.SetValue(null, null); - } - - [TestCase("Square Bracket Columnizer", "30/08/2018 08:51:42.712 [TRACE] [a] hello", "30/08/2018 08:51:42.712 [DATAIO] [b] world", null, null, null)] - [TestCase("Square Bracket Columnizer", "30/08/2018 08:51:42.712 [TRACE] hello", "30/08/2018 08:51:42.712 [DATAIO][] world", null, null, null)] - [TestCase("Square Bracket Columnizer", "", "30/08/2018 08:51:42.712 [TRACE] hello", "30/08/2018 08:51:42.712 [TRACE] hello", "[DATAIO][b][c] world", null)] - [TestCase("Timestamp Columnizer", "30/08/2018 08:51:42.712 no bracket 1", "30/08/2018 08:51:42.712 no bracket 2", "30/08/2018 08:51:42.712 [TRACE] with bracket 1", "30/08/2018 08:51:42.712 [TRACE] with bracket 2", "no bracket 3")] - public void FindColumnizer_ReturnCorrectColumnizer (string expectedColumnizerName, string line0, string line1, string line2, string line3, string line4) - { - var path = Path.Join(AppDomain.CurrentDomain.BaseDirectory, "test"); - - Mock autoLogLineColumnizerCallbackMock = new(); - - // Mock GetLogLineMemory() which returns ILogLineMemory - _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(0)).Returns(new TestLogLineMemory() - { - FullLine = line0?.AsMemory() ?? ReadOnlyMemory.Empty, - LineNumber = 0 - }); - - _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(1)).Returns(new TestLogLineMemory() - { - FullLine = line1?.AsMemory() ?? ReadOnlyMemory.Empty, - LineNumber = 1 - }); - - _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(2)).Returns(new TestLogLineMemory() - { - FullLine = line2?.AsMemory() ?? ReadOnlyMemory.Empty, - LineNumber = 2 - }); - - _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(3)).Returns(new TestLogLineMemory() - { - FullLine = line3?.AsMemory() ?? ReadOnlyMemory.Empty, - LineNumber = 3 - }); - - _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(4)).Returns(new TestLogLineMemory() - { - FullLine = line4?.AsMemory() ?? ReadOnlyMemory.Empty, - LineNumber = 4 - }); - - // Mock for additional sampled lines that ColumnizerPicker checks - _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(5)).Returns((ILogLineMemory)null); - _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(25)).Returns((ILogLineMemory)null); - _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(100)).Returns((ILogLineMemory)null); - _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(200)).Returns((ILogLineMemory)null); - _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(400)).Returns((ILogLineMemory)null); - - var result = ColumnizerPicker.FindMemoryColumnizer(path, autoLogLineColumnizerCallbackMock.Object, PluginRegistry.PluginRegistry.Instance.RegisteredColumnizers); - - Assert.That(result.GetName(), Is.EqualTo(expectedColumnizerName)); - } - - [TestCase(@".\TestData\JsonColumnizerTest_01.txt", typeof(JsonCompactColumnizer.JsonCompactColumnizer), ReaderType.System)] - [TestCase(@".\TestData\SquareBracketColumnizerTest_02.txt", typeof(SquareBracketColumnizer), ReaderType.System)] - public void FindReplacementForAutoColumnizer_ValidTextFile_ReturnCorrectColumnizer (string fileName, Type columnizerType, ReaderType readerType) - { - var pluginRegistry = PluginRegistry.PluginRegistry.Instance; - - var path = Path.Join(AppDomain.CurrentDomain.BaseDirectory, fileName); - LogfileReader reader = new(path, new EncodingOptions(), true, 40, 50, new MultiFileOptions(), readerType, pluginRegistry, 500); - reader.ReadFiles(); - - Mock autoColumnizer = new(); - _ = autoColumnizer.Setup(a => a.GetName()).Returns("Auto Columnizer"); - - // TODO: When DI container is ready, we can mock this set up. - PluginRegistry.PluginRegistry.Instance.RegisteredColumnizers.Add(new JsonCompactColumnizer.JsonCompactColumnizer()); - var result = ColumnizerPicker.FindReplacementForAutoMemoryColumnizer(fileName, reader, autoColumnizer.Object, PluginRegistry.PluginRegistry.Instance.RegisteredColumnizers); - - Assert.That(columnizerType, Is.EqualTo(result.GetType())); - } - - [TestCase(@".\TestData\FileNotExists.txt", typeof(DefaultLogfileColumnizer))] - public void DecideColumnizerByName_WhenReaderIsNotReady_ReturnCorrectColumnizer (string fileName, Type columnizerType) - { - // TODO: When DI container is ready, we can mock this set up. - PluginRegistry.PluginRegistry.Instance.RegisteredColumnizers.Add(new JsonCompactColumnizer.JsonCompactColumnizer()); - var result = ColumnizerPicker.DecideMemoryColumnizerByName(fileName, PluginRegistry.PluginRegistry.Instance.RegisteredColumnizers); - - Assert.That(columnizerType, Is.EqualTo(result.GetType())); - } - - [TestCase(@"Invalid Name", typeof(DefaultLogfileColumnizer))] - [TestCase(@"JSON Columnizer", typeof(JsonColumnizer.JsonColumnizer))] - public void DecideColumnizerByName_ValidTextFile_ReturnCorrectColumnizer (string columnizerName, Type columnizerType) - { - // TODO: When DI container is ready, we can mock this set up. - PluginRegistry.PluginRegistry.Instance.RegisteredColumnizers.Add(new JsonColumnizer.JsonColumnizer()); - - var result = ColumnizerPicker.DecideMemoryColumnizerByName(columnizerName, PluginRegistry.PluginRegistry.Instance.RegisteredColumnizers); - - Assert.That(columnizerType, Is.EqualTo(result.GetType())); - } - - /// - /// Test helper class that implements ILogLineMemory for mocking log lines. - /// - private class TestLogLineMemory : ILogLineMemory - { - public ReadOnlyMemory FullLine { get; set; } - - public int LineNumber { get; set; } - - // Explicit implementation for ITextValueMemory.Text (ReadOnlyMemory version) - ReadOnlyMemory ITextValueMemory.Text => FullLine; - } +using System.Reflection; + +using ColumnizerLib; + +using LogExpert.Core.Classes.Columnizer; +using LogExpert.Core.Classes.Log; +using LogExpert.Core.Entities; +using LogExpert.Core.Enums; + +using Moq; + +using NUnit.Framework; + +namespace LogExpert.Tests.ColumnizerTests; + +/// +/// Summary description for AutoColumnizerTest +/// +[TestFixture] +public class ColumnizerPickerTest +{ + [SetUp] + public void Setup () + { + // Reset singleton for testing (same pattern as PluginRegistryTests) + ResetPluginRegistrySingleton(); + + // Initialize plugin registry with proper test directory + var testDataPath = Path.Join(Path.GetTempPath(), "LogExpertTests", Guid.NewGuid().ToString()); + _ = Directory.CreateDirectory(testDataPath); + + var pluginRegistry = PluginRegistry.PluginRegistry.Create(testDataPath, 250); + + // Verify the local file system plugin is registered + var localPlugin = pluginRegistry.FindFileSystemForUri(@"C:\test.txt"); + Assert.That(localPlugin, Is.Not.Null, "Local file system plugin not registered!"); + } + + [TearDown] + public void TearDown () + { + ResetPluginRegistrySingleton(); + } + + /// + /// Uses reflection to reset the singleton instance for testing. + /// This ensures each test starts with a fresh PluginRegistry state. + /// + private static void ResetPluginRegistrySingleton () + { + var instanceField = typeof(PluginRegistry.PluginRegistry).GetField("_instance", BindingFlags.Static | BindingFlags.NonPublic); + instanceField?.SetValue(null, null); + } + + [TestCase("Square Bracket Columnizer", "30/08/2018 08:51:42.712 [TRACE] [a] hello", "30/08/2018 08:51:42.712 [DATAIO] [b] world", null, null, null)] + [TestCase("Square Bracket Columnizer", "30/08/2018 08:51:42.712 [TRACE] hello", "30/08/2018 08:51:42.712 [DATAIO][] world", null, null, null)] + [TestCase("Square Bracket Columnizer", "", "30/08/2018 08:51:42.712 [TRACE] hello", "30/08/2018 08:51:42.712 [TRACE] hello", "[DATAIO][b][c] world", null)] + [TestCase("Timestamp Columnizer", "30/08/2018 08:51:42.712 no bracket 1", "30/08/2018 08:51:42.712 no bracket 2", "30/08/2018 08:51:42.712 [TRACE] with bracket 1", "30/08/2018 08:51:42.712 [TRACE] with bracket 2", "no bracket 3")] + public void FindColumnizer_ReturnCorrectColumnizer (string expectedColumnizerName, string line0, string line1, string line2, string line3, string line4) + { + var path = Path.Join(AppDomain.CurrentDomain.BaseDirectory, "test"); + + Mock autoLogLineColumnizerCallbackMock = new(); + + // Mock GetLogLineMemory() which returns ILogLineMemory + _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(0)).Returns(new TestLogLineMemory() + { + FullLine = line0?.AsMemory() ?? ReadOnlyMemory.Empty, + LineNumber = 0 + }); + + _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(1)).Returns(new TestLogLineMemory() + { + FullLine = line1?.AsMemory() ?? ReadOnlyMemory.Empty, + LineNumber = 1 + }); + + _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(2)).Returns(new TestLogLineMemory() + { + FullLine = line2?.AsMemory() ?? ReadOnlyMemory.Empty, + LineNumber = 2 + }); + + _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(3)).Returns(new TestLogLineMemory() + { + FullLine = line3?.AsMemory() ?? ReadOnlyMemory.Empty, + LineNumber = 3 + }); + + _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(4)).Returns(new TestLogLineMemory() + { + FullLine = line4?.AsMemory() ?? ReadOnlyMemory.Empty, + LineNumber = 4 + }); + + // Mock for additional sampled lines that ColumnizerPicker checks + _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(5)).Returns((ILogLineMemory)null); + _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(25)).Returns((ILogLineMemory)null); + _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(100)).Returns((ILogLineMemory)null); + _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(200)).Returns((ILogLineMemory)null); + _ = autoLogLineColumnizerCallbackMock.Setup(a => a.GetLogLineMemory(400)).Returns((ILogLineMemory)null); + + var result = ColumnizerPicker.FindMemoryColumnizer(path, autoLogLineColumnizerCallbackMock.Object, PluginRegistry.PluginRegistry.Instance.RegisteredColumnizers); + + Assert.That(result.GetName(), Is.EqualTo(expectedColumnizerName)); + } + + [TestCase(@".\TestData\JsonColumnizerTest_01.txt", typeof(JsonCompactColumnizer.JsonCompactColumnizer), ReaderType.System)] + [TestCase(@".\TestData\SquareBracketColumnizerTest_02.txt", typeof(SquareBracketColumnizer), ReaderType.System)] + public void FindReplacementForAutoColumnizer_ValidTextFile_ReturnCorrectColumnizer (string fileName, Type columnizerType, ReaderType readerType) + { + var pluginRegistry = PluginRegistry.PluginRegistry.Instance; + + var path = Path.Join(AppDomain.CurrentDomain.BaseDirectory, fileName); + LogfileReader reader = new(path, new EncodingOptions(), true, 40, 50, new MultiFileOptions(), readerType, pluginRegistry, 500); + reader.ReadFiles(); + + Mock autoColumnizer = new(); + _ = autoColumnizer.Setup(a => a.GetName()).Returns("Auto Columnizer"); + + // TODO: When DI container is ready, we can mock this set up. + PluginRegistry.PluginRegistry.Instance.RegisteredColumnizers.Add(new JsonCompactColumnizer.JsonCompactColumnizer()); + var result = ColumnizerPicker.FindReplacementForAutoMemoryColumnizer(fileName, reader, autoColumnizer.Object, PluginRegistry.PluginRegistry.Instance.RegisteredColumnizers); + + Assert.That(columnizerType, Is.EqualTo(result.GetType())); + } + + [TestCase(@".\TestData\FileNotExists.txt", typeof(DefaultLogfileColumnizer))] + public void DecideColumnizerByName_WhenReaderIsNotReady_ReturnCorrectColumnizer (string fileName, Type columnizerType) + { + // TODO: When DI container is ready, we can mock this set up. + PluginRegistry.PluginRegistry.Instance.RegisteredColumnizers.Add(new JsonCompactColumnizer.JsonCompactColumnizer()); + var result = ColumnizerPicker.DecideMemoryColumnizerByName(fileName, PluginRegistry.PluginRegistry.Instance.RegisteredColumnizers); + + Assert.That(columnizerType, Is.EqualTo(result.GetType())); + } + + [TestCase(@"Invalid Name", typeof(DefaultLogfileColumnizer))] + [TestCase(@"JSON Columnizer", typeof(JsonColumnizer.JsonColumnizer))] + public void DecideColumnizerByName_ValidTextFile_ReturnCorrectColumnizer (string columnizerName, Type columnizerType) + { + // TODO: When DI container is ready, we can mock this set up. + PluginRegistry.PluginRegistry.Instance.RegisteredColumnizers.Add(new JsonColumnizer.JsonColumnizer()); + + var result = ColumnizerPicker.DecideMemoryColumnizerByName(columnizerName, PluginRegistry.PluginRegistry.Instance.RegisteredColumnizers); + + Assert.That(columnizerType, Is.EqualTo(result.GetType())); + } + + /// + /// Test helper class that implements ILogLineMemory for mocking log lines. + /// + private class TestLogLineMemory : ILogLineMemory + { + public ReadOnlyMemory FullLine { get; set; } + + public int LineNumber { get; set; } + + // Explicit implementation for ITextValueMemory.Text (ReadOnlyMemory version) + ReadOnlyMemory ITextValueMemory.Text => FullLine; + } } \ No newline at end of file diff --git a/src/LogExpert.Tests/JsonColumnizerTest.cs b/src/LogExpert.Tests/ColumnizerTests/JsonColumnizerTest.cs similarity index 98% rename from src/LogExpert.Tests/JsonColumnizerTest.cs rename to src/LogExpert.Tests/ColumnizerTests/JsonColumnizerTest.cs index 46fb007d..fc344da5 100644 --- a/src/LogExpert.Tests/JsonColumnizerTest.cs +++ b/src/LogExpert.Tests/ColumnizerTests/JsonColumnizerTest.cs @@ -6,7 +6,7 @@ using NUnit.Framework; -namespace LogExpert.Tests; +namespace LogExpert.Tests.ColumnizerTests; [TestFixture] public class JsonColumnizerTest diff --git a/src/LogExpert.Tests/JsonCompactColumnizerTest.cs b/src/LogExpert.Tests/ColumnizerTests/JsonCompactColumnizerTest.cs similarity index 96% rename from src/LogExpert.Tests/JsonCompactColumnizerTest.cs rename to src/LogExpert.Tests/ColumnizerTests/JsonCompactColumnizerTest.cs index 2ac88e03..e16118ab 100644 --- a/src/LogExpert.Tests/JsonCompactColumnizerTest.cs +++ b/src/LogExpert.Tests/ColumnizerTests/JsonCompactColumnizerTest.cs @@ -1,77 +1,77 @@ -using System.Reflection; - -using ColumnizerLib; - -using LogExpert.Core.Classes.Log; -using LogExpert.Core.Entities; -using LogExpert.Core.Enums; - -using NUnit.Framework; - -namespace LogExpert.Tests; - -[TestFixture] -public class JsonCompactColumnizerTest -{ - [SetUp] - public void Setup () - { - // Reset singleton for testing (same pattern as PluginRegistryTests) - ResetPluginRegistrySingleton(); - - // Initialize plugin registry with proper test directory - var testDataPath = Path.Join(Path.GetTempPath(), "LogExpertTests", Guid.NewGuid().ToString()); - _ = Directory.CreateDirectory(testDataPath); - - var pluginRegistry = PluginRegistry.PluginRegistry.Create(testDataPath, 250); - - // Verify the local file system plugin is registered - var localPlugin = pluginRegistry.FindFileSystemForUri(@"C:\test.txt"); - Assert.That(localPlugin, Is.Not.Null, "Local file system plugin not registered!"); - } - - [TearDown] - public void TearDown () - { - ResetPluginRegistrySingleton(); - } - - /// - /// Uses reflection to reset the singleton instance for testing. - /// This ensures each test starts with a fresh PluginRegistry state. - /// - private static void ResetPluginRegistrySingleton () - { - var instanceField = typeof(PluginRegistry.PluginRegistry).GetField("_instance", BindingFlags.Static | BindingFlags.NonPublic); - instanceField?.SetValue(null, null); - } - - // As long as the json file contains one of the pre-defined key, it's perfectly supported. - [TestCase(@".\TestData\JsonCompactColumnizerTest_01.json", Priority.PerfectlySupport, ReaderType.System)] - [TestCase(@".\TestData\JsonCompactColumnizerTest_02.json", Priority.PerfectlySupport, ReaderType.System)] - [TestCase(@".\TestData\JsonCompactColumnizerTest_03.json", Priority.WellSupport, ReaderType.System)] - public void GetPriority_HappyFile_PriorityMatches (string fileName, Priority priority, ReaderType readerType) - { - var jsonCompactColumnizer = new JsonCompactColumnizer.JsonCompactColumnizer(); - var path = Path.Join(AppDomain.CurrentDomain.BaseDirectory, fileName); - LogfileReader logFileReader = new(path, new EncodingOptions(), true, 40, 50, new MultiFileOptions(), readerType, PluginRegistry.PluginRegistry.Instance, 500); - logFileReader.ReadFiles(); - List loglines = - [ - // Sampling a few lines to select the correct columnizer - logFileReader.GetLogLineMemory(0), - logFileReader.GetLogLineMemory(1), - logFileReader.GetLogLineMemory(2), - logFileReader.GetLogLineMemory(3), - logFileReader.GetLogLineMemory(4), - logFileReader.GetLogLineMemory(5), - logFileReader.GetLogLineMemory(25), - logFileReader.GetLogLineMemory(100), - logFileReader.GetLogLineMemory(200), - logFileReader.GetLogLineMemory(400) - ]; - - var result = jsonCompactColumnizer.GetPriority(path, loglines); - Assert.That(result, Is.EqualTo(priority)); - } +using System.Reflection; + +using ColumnizerLib; + +using LogExpert.Core.Classes.Log; +using LogExpert.Core.Entities; +using LogExpert.Core.Enums; + +using NUnit.Framework; + +namespace LogExpert.Tests.ColumnizerTests; + +[TestFixture] +public class JsonCompactColumnizerTest +{ + [SetUp] + public void Setup () + { + // Reset singleton for testing (same pattern as PluginRegistryTests) + ResetPluginRegistrySingleton(); + + // Initialize plugin registry with proper test directory + var testDataPath = Path.Join(Path.GetTempPath(), "LogExpertTests", Guid.NewGuid().ToString()); + _ = Directory.CreateDirectory(testDataPath); + + var pluginRegistry = PluginRegistry.PluginRegistry.Create(testDataPath, 250); + + // Verify the local file system plugin is registered + var localPlugin = pluginRegistry.FindFileSystemForUri(@"C:\test.txt"); + Assert.That(localPlugin, Is.Not.Null, "Local file system plugin not registered!"); + } + + [TearDown] + public void TearDown () + { + ResetPluginRegistrySingleton(); + } + + /// + /// Uses reflection to reset the singleton instance for testing. + /// This ensures each test starts with a fresh PluginRegistry state. + /// + private static void ResetPluginRegistrySingleton () + { + var instanceField = typeof(PluginRegistry.PluginRegistry).GetField("_instance", BindingFlags.Static | BindingFlags.NonPublic); + instanceField?.SetValue(null, null); + } + + // As long as the json file contains one of the pre-defined key, it's perfectly supported. + [TestCase(@".\TestData\JsonCompactColumnizerTest_01.json", Priority.PerfectlySupport, ReaderType.System)] + [TestCase(@".\TestData\JsonCompactColumnizerTest_02.json", Priority.PerfectlySupport, ReaderType.System)] + [TestCase(@".\TestData\JsonCompactColumnizerTest_03.json", Priority.WellSupport, ReaderType.System)] + public void GetPriority_HappyFile_PriorityMatches (string fileName, Priority priority, ReaderType readerType) + { + var jsonCompactColumnizer = new JsonCompactColumnizer.JsonCompactColumnizer(); + var path = Path.Join(AppDomain.CurrentDomain.BaseDirectory, fileName); + LogfileReader logFileReader = new(path, new EncodingOptions(), true, 40, 50, new MultiFileOptions(), readerType, PluginRegistry.PluginRegistry.Instance, 500); + logFileReader.ReadFiles(); + List loglines = + [ + // Sampling a few lines to select the correct columnizer + logFileReader.GetLogLineMemory(0), + logFileReader.GetLogLineMemory(1), + logFileReader.GetLogLineMemory(2), + logFileReader.GetLogLineMemory(3), + logFileReader.GetLogLineMemory(4), + logFileReader.GetLogLineMemory(5), + logFileReader.GetLogLineMemory(25), + logFileReader.GetLogLineMemory(100), + logFileReader.GetLogLineMemory(200), + logFileReader.GetLogLineMemory(400) + ]; + + var result = jsonCompactColumnizer.GetPriority(path, loglines); + Assert.That(result, Is.EqualTo(priority)); + } } \ No newline at end of file diff --git a/src/LogExpert.Tests/SquareBracketColumnizerTest.cs b/src/LogExpert.Tests/ColumnizerTests/SquareBracketColumnizerTest.cs similarity index 98% rename from src/LogExpert.Tests/SquareBracketColumnizerTest.cs rename to src/LogExpert.Tests/ColumnizerTests/SquareBracketColumnizerTest.cs index 1b6f017f..01f99022 100644 --- a/src/LogExpert.Tests/SquareBracketColumnizerTest.cs +++ b/src/LogExpert.Tests/ColumnizerTests/SquareBracketColumnizerTest.cs @@ -9,7 +9,7 @@ using NUnit.Framework; -namespace LogExpert.Tests; +namespace LogExpert.Tests.ColumnizerTests; [TestFixture] public class SquareBracketColumnizerTest diff --git a/src/LogExpert.Tests/ConfigManagerPortableModeTests.cs b/src/LogExpert.Tests/ConfigManagerTests/ConfigManagerPortableModeTests.cs similarity index 96% rename from src/LogExpert.Tests/ConfigManagerPortableModeTests.cs rename to src/LogExpert.Tests/ConfigManagerTests/ConfigManagerPortableModeTests.cs index 4f7a3206..bb98741f 100644 --- a/src/LogExpert.Tests/ConfigManagerPortableModeTests.cs +++ b/src/LogExpert.Tests/ConfigManagerTests/ConfigManagerPortableModeTests.cs @@ -1,10 +1,8 @@ using System.Reflection; -using LogExpert.Configuration; - using NUnit.Framework; -namespace LogExpert.Tests; +namespace LogExpert.Tests.ConfigManagerTests; /// /// Unit tests for ConfigManager portable mode functionality. @@ -15,7 +13,7 @@ namespace LogExpert.Tests; public class ConfigManagerPortableModeTests { private string _testDir; - private ConfigManager _configManager; + private Configuration.ConfigManager _configManager; [SetUp] public void SetUp () @@ -25,7 +23,7 @@ public void SetUp () _ = Directory.CreateDirectory(_testDir); // Initialize ConfigManager for testing - _configManager = ConfigManager.Instance; + _configManager = Configuration.ConfigManager.Instance; // Reset the singleton's initialization state using reflection ResetConfigManagerInitialization(); @@ -71,11 +69,11 @@ public void TearDown () /// private void ResetConfigManagerInitialization () { - var isInitializedField = typeof(ConfigManager).GetField("_isInitialized", BindingFlags.NonPublic | BindingFlags.Instance); + var isInitializedField = typeof(Configuration.ConfigManager).GetField("_isInitialized", BindingFlags.NonPublic | BindingFlags.Instance); isInitializedField?.SetValue(_configManager, false); // Reset settings so they reload from the new path - var settingsField = typeof(ConfigManager).GetField("_settings", BindingFlags.NonPublic | BindingFlags.Instance); + var settingsField = typeof(Configuration.ConfigManager).GetField("_settings", BindingFlags.NonPublic | BindingFlags.Instance); settingsField?.SetValue(_configManager, null); } @@ -85,7 +83,7 @@ private void ResetConfigManagerInitialization () [System.Diagnostics.CodeAnalysis.SuppressMessage("Usage", "CA2201:Do not raise reserved exception types", Justification = "Unit Tests")] private void InvokePrivateInstanceMethod (string methodName, params object[] parameters) { - MethodInfo? method = typeof(ConfigManager).GetMethod(methodName, BindingFlags.NonPublic | BindingFlags.Instance) + MethodInfo? method = typeof(Configuration.ConfigManager).GetMethod(methodName, BindingFlags.NonPublic | BindingFlags.Instance) ?? throw new Exception($"Instance method {methodName} not found"); _ = method.Invoke(_configManager, parameters); @@ -654,7 +652,7 @@ public void MoveFileIfExists_MovesFile () File.WriteAllText(source, "test content"); // Act - var method = typeof(ConfigManager).GetMethod("MoveFileIfExists", BindingFlags.NonPublic | BindingFlags.Static); + var method = typeof(Configuration.ConfigManager).GetMethod("MoveFileIfExists", BindingFlags.NonPublic | BindingFlags.Static); _ = (method?.Invoke(null, [source, target])); // Assert @@ -673,7 +671,7 @@ public void MoveFileIfExists_NonExistentSource_DoesNothing () var target = Path.Join(_testDir, "target.txt"); // Act - var method = typeof(ConfigManager).GetMethod("MoveFileIfExists", BindingFlags.NonPublic | BindingFlags.Static); + var method = typeof(Configuration.ConfigManager).GetMethod("MoveFileIfExists", BindingFlags.NonPublic | BindingFlags.Static); _ = (method?.Invoke(null, [source, target])); // Assert @@ -691,7 +689,7 @@ public void CopyFileIfExists_CopiesFile () File.WriteAllText(source, "copy me"); // Act - var method = typeof(ConfigManager).GetMethod("CopyFileIfExists", BindingFlags.NonPublic | BindingFlags.Static); + var method = typeof(Configuration.ConfigManager).GetMethod("CopyFileIfExists", BindingFlags.NonPublic | BindingFlags.Static); _ = (method?.Invoke(null, [source, target])); // Assert @@ -712,7 +710,7 @@ public void CopyFileIfNotExists_DoesNotOverwrite () File.WriteAllText(target, "existing content"); // Act - var method = typeof(ConfigManager).GetMethod("CopyFileIfNotExists", BindingFlags.NonPublic | BindingFlags.Static); + var method = typeof(Configuration.ConfigManager).GetMethod("CopyFileIfNotExists", BindingFlags.NonPublic | BindingFlags.Static); _ = (method?.Invoke(null, [source, target])); //Assert @@ -736,7 +734,7 @@ public void CopyDirectoryRecursive_CopiesEverything () File.WriteAllText(Path.Join(subDir, "file2.txt"), "content2"); // Act - var method = typeof(ConfigManager).GetMethod("CopyDirectoryRecursive", BindingFlags.NonPublic | BindingFlags.Static); + var method = typeof(Configuration.ConfigManager).GetMethod("CopyDirectoryRecursive", BindingFlags.NonPublic | BindingFlags.Static); _ = (method?.Invoke(null, [sourceDir, targetDir])); // Assert diff --git a/src/LogExpert.Tests/ConfigManagerTest.cs b/src/LogExpert.Tests/ConfigManagerTests/ConfigManagerTest.cs similarity index 94% rename from src/LogExpert.Tests/ConfigManagerTest.cs rename to src/LogExpert.Tests/ConfigManagerTests/ConfigManagerTest.cs index 845b5489..275a1fb8 100644 --- a/src/LogExpert.Tests/ConfigManagerTest.cs +++ b/src/LogExpert.Tests/ConfigManagerTests/ConfigManagerTest.cs @@ -1,6 +1,5 @@ using System.Reflection; -using LogExpert.Configuration; using LogExpert.Core.Classes.Filter; using LogExpert.Core.Config; using LogExpert.Core.Entities; @@ -9,7 +8,7 @@ using NUnit.Framework; -namespace LogExpert.Tests; +namespace LogExpert.Tests.ConfigManagerTests; /// /// Unit tests for ConfigManager settings loss prevention fixes. @@ -20,10 +19,9 @@ public class ConfigManagerTest { private string _testDir; private FileInfo _testSettingsFile; - private ConfigManager _configManager; + private Configuration.ConfigManager _configManager; [SetUp] - [System.Diagnostics.CodeAnalysis.SuppressMessage("Interoperability", "CA1416:Validate platform compatibility", Justification = "Unit Test")] public void SetUp () { // Create isolated test directory for each test @@ -32,7 +30,7 @@ public void SetUp () _testSettingsFile = new FileInfo(Path.Join(_testDir, "settings.json")); // Initialize ConfigManager for testing - _configManager = ConfigManager.Instance; + _configManager = Configuration.ConfigManager.Instance; _configManager.Initialize(_testDir, new Rectangle(0, 0, 1920, 1080)); } @@ -65,7 +63,7 @@ public void TearDown () [System.Diagnostics.CodeAnalysis.SuppressMessage("Usage", "CA2201:Do not raise reserved exception types", Justification = "Unit Tests")] private static T InvokePrivateStaticMethod (string methodName, params object[] parameters) { - MethodInfo? method = typeof(ConfigManager).GetMethod(methodName, BindingFlags.NonPublic | BindingFlags.Static); + MethodInfo? method = typeof(Configuration.ConfigManager).GetMethod(methodName, BindingFlags.NonPublic | BindingFlags.Static); return method == null ? throw new Exception($"Static method {methodName} not found") @@ -78,7 +76,7 @@ private static T InvokePrivateStaticMethod (string methodName, params object[ [System.Diagnostics.CodeAnalysis.SuppressMessage("Usage", "CA2201:Do not raise reserved exception types", Justification = "Unit Tests")] private T InvokePrivateInstanceMethod (string methodName, params object[] parameters) { - MethodInfo? method = typeof(ConfigManager).GetMethod(methodName, BindingFlags.NonPublic | BindingFlags.Instance); + MethodInfo? method = typeof(Configuration.ConfigManager).GetMethod(methodName, BindingFlags.NonPublic | BindingFlags.Instance); return method == null ? throw new Exception($"Instance method {methodName} not found") @@ -91,7 +89,7 @@ private T InvokePrivateInstanceMethod (string methodName, params object[] par [System.Diagnostics.CodeAnalysis.SuppressMessage("Usage", "CA2201:Do not raise reserved exception types", Justification = "Unit Tests")] private void InvokePrivateInstanceMethod (string methodName, params object[] parameters) { - MethodInfo? method = typeof(ConfigManager).GetMethod(methodName, BindingFlags.NonPublic | BindingFlags.Instance) + MethodInfo? method = typeof(Configuration.ConfigManager).GetMethod(methodName, BindingFlags.NonPublic | BindingFlags.Instance) ?? throw new Exception($"Instance method {methodName} not found"); _ = method.Invoke(_configManager, parameters); @@ -553,7 +551,6 @@ public void LoadOrCreateNew_InvalidJSON_HandlesGracefully () [Test] [Category("Import")] [Description("Import should handle null _settings field by using Settings property")] - [System.Diagnostics.CodeAnalysis.SuppressMessage("Interoperability", "CA1416:Validate platform compatibility", Justification = "Unit Test")] [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Test")] public void Import_WithUninitializedSettings_ShouldNotThrowNullReference () { @@ -578,7 +575,6 @@ public void Import_WithUninitializedSettings_ShouldNotThrowNullReference () [Test] [Category("Import")] [Description("Import should validate that import file exists")] - [System.Diagnostics.CodeAnalysis.SuppressMessage("Interoperability", "CA1416:Validate platform compatibility", Justification = "Unit Test")] public void Import_WithNonExistentFile_ShouldReturnFailure () { // Arrange @@ -596,7 +592,6 @@ public void Import_WithNonExistentFile_ShouldReturnFailure () [Test] [Category("Import")] [Description("Import should validate that import file is not null")] - [System.Diagnostics.CodeAnalysis.SuppressMessage("Interoperability", "CA1416:Validate platform compatibility", Justification = "Unit Test")] public void Import_WithNullFileInfo_ShouldReturnFailure () { // Act @@ -611,7 +606,6 @@ public void Import_WithNullFileInfo_ShouldReturnFailure () [Test] [Category("Import")] [Description("Import should detect corrupted import files")] - [System.Diagnostics.CodeAnalysis.SuppressMessage("Interoperability", "CA1416:Validate platform compatibility", Justification = "Unit Test")] public void Import_WithCorruptedFile_ShouldReturnFailure () { // Arrange @@ -631,7 +625,6 @@ public void Import_WithCorruptedFile_ShouldReturnFailure () [Test] [Category("Import")] [Description("Import should detect empty/default settings and require confirmation")] - [System.Diagnostics.CodeAnalysis.SuppressMessage("Interoperability", "CA1416:Validate platform compatibility", Justification = "Unit Test")] public void Import_WithEmptySettings_ShouldRequireConfirmation () { // Arrange @@ -652,7 +645,6 @@ public void Import_WithEmptySettings_ShouldRequireConfirmation () [Test] [Category("Import")] [Description("Import should successfully import valid populated settings")] - [System.Diagnostics.CodeAnalysis.SuppressMessage("Interoperability", "CA1416:Validate platform compatibility", Justification = "Unit Test")] public void Import_WithValidPopulatedSettings_ShouldSucceed () { // Arrange @@ -681,7 +673,6 @@ public void Import_WithValidPopulatedSettings_ShouldSucceed () [Test] [Category("Import")] [Description("Import with Other flag should merge preferences correctly")] - [System.Diagnostics.CodeAnalysis.SuppressMessage("Interoperability", "CA1416:Validate platform compatibility", Justification = "Unit Test")] public void Import_WithOtherFlag_ShouldMergePreferences () { // Arrange @@ -713,7 +704,6 @@ public void Import_WithOtherFlag_ShouldMergePreferences () [Test] [Category("Import")] [Description("Import with ColumnizerMasks flag should import columnizer masks")] - [System.Diagnostics.CodeAnalysis.SuppressMessage("Interoperability", "CA1416:Validate platform compatibility", Justification = "Unit Test")] public void Import_WithColumnizerMasksFlag_ShouldImportMasks () { // Arrange @@ -739,7 +729,6 @@ public void Import_WithColumnizerMasksFlag_ShouldImportMasks () [Test] [Category("Import")] [Description("Import with KeepExisting flag should merge rather than replace")] - [System.Diagnostics.CodeAnalysis.SuppressMessage("Interoperability", "CA1416:Validate platform compatibility", Justification = "Unit Test")] public void Import_WithKeepExistingFlag_ShouldMergeSettings () { // Arrange @@ -771,7 +760,6 @@ public void Import_WithKeepExistingFlag_ShouldMergeSettings () [Test] [Category("Import")] [Description("Import should handle null Preferences in import file gracefully")] - [System.Diagnostics.CodeAnalysis.SuppressMessage("Interoperability", "CA1416:Validate platform compatibility", Justification = "Unit Test")] [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Test")] public void Import_WithNullPreferences_ShouldHandleGracefully () { @@ -797,7 +785,6 @@ public void Import_WithNullPreferences_ShouldHandleGracefully () [Test] [Category("Import")] [Description("Multiple imports should maintain consistency")] - [System.Diagnostics.CodeAnalysis.SuppressMessage("Interoperability", "CA1416:Validate platform compatibility", Justification = "Unit Test")] public void Import_MultipleImports_ShouldMaintainConsistency () { // Arrange & Act - Multiple imports @@ -823,7 +810,6 @@ public void Import_MultipleImports_ShouldMaintainConsistency () [Test] [Category("Import")] [Description("Import should save settings after successful import")] - [System.Diagnostics.CodeAnalysis.SuppressMessage("Interoperability", "CA1416:Validate platform compatibility", Justification = "Unit Test")] public void Import_SuccessfulImport_ShouldSaveSettings () { // Arrange diff --git a/src/LogExpert.Tests/LogStreamReaderTest.cs b/src/LogExpert.Tests/LogStreamReaderTest.cs deleted file mode 100644 index bf39b832..00000000 --- a/src/LogExpert.Tests/LogStreamReaderTest.cs +++ /dev/null @@ -1,247 +0,0 @@ -using System.Text; - -using LogExpert.Core.Classes.Log; -using LogExpert.Core.Entities; - -using NUnit.Framework; - -namespace LogExpert.Tests; - -[TestFixture] -public class LogStreamReaderTest -{ - [Test] - [TestCase("Line 1\nLine 2\nLine 3", 3)] - [TestCase("Line 1\nLine 2\nLine 3\n", 3)] - [TestCase("Line 1\r\nLine 2\r\nLine 3", 3)] - [TestCase("Line 1\r\nLine 2\r\nLine 3\r\n", 3)] - [TestCase("Line 1\rLine 2\rLine 3", 3)] - [TestCase("Line 1\rLine 2\rLine 3\r", 3)] - public void ReadLinesWithSystemNewLine (string text, int expectedLines) - { - using var stream = new MemoryStream(Encoding.ASCII.GetBytes(text)); - using var reader = new PositionAwareStreamReaderSystem(stream, new EncodingOptions(), 500); - var lineCount = 0; - while (true) - { - var line = reader.ReadLine(); - if (line == null) - { - break; - } - - lineCount += 1; - - Assert.That(line.StartsWith($"Line {lineCount}", StringComparison.OrdinalIgnoreCase), $"Invalid line: {line}"); - } - - Assert.That(expectedLines, Is.EqualTo(lineCount), $"Unexpected lines:\n{text}"); - } - - [Test] - [TestCase("\n\n\n", 3)] - [TestCase("\r\n\r\n\r\n", 3)] - [TestCase("\r\r\r", 3)] - public void CountLinesWithSystemNewLine (string text, int expectedLines) - { - using var stream = new MemoryStream(Encoding.ASCII.GetBytes(text)); - using var reader = new PositionAwareStreamReaderSystem(stream, new EncodingOptions(), 500); - var lineCount = 0; - while (reader.ReadLine() != null) - { - lineCount += 1; - } - - Assert.That(expectedLines, Is.EqualTo(lineCount), $"Unexpected lines:\n{text}"); - } - - [Test] - [TestCase("Line 1\nLine 2\nLine 3", 3)] - [TestCase("Line 1\nLine 2\nLine 3\n", 3)] - [TestCase("Line 1\r\nLine 2\r\nLine 3", 3)] - [TestCase("Line 1\r\nLine 2\r\nLine 3\r\n", 3)] - [TestCase("Line 1\rLine 2\rLine 3", 3)] - [TestCase("Line 1\rLine 2\rLine 3\r", 3)] - public void ReadLinesWithLegacyNewLine (string text, int expectedLines) - { - using var stream = new MemoryStream(Encoding.ASCII.GetBytes(text)); - using var reader = new PositionAwareStreamReaderLegacy(stream, new EncodingOptions(), 500); - var lineCount = 0; - while (true) - { - var line = reader.ReadLine(); - if (line == null) - { - break; - } - - lineCount += 1; - - Assert.That(line.StartsWith($"Line {lineCount}", StringComparison.OrdinalIgnoreCase), $"Invalid line: {line}"); - } - - Assert.That(expectedLines, Is.EqualTo(lineCount), $"Unexpected lines:\n{text}"); - } - [Test] - [TestCase("\n\n\n", 3)] - [TestCase("\r\n\r\n\r\n", 3)] - [TestCase("\r\r\r", 3)] - public void CountLinesWithLegacyNewLine (string text, int expectedLines) - { - using var stream = new MemoryStream(Encoding.ASCII.GetBytes(text)); - using var reader = new PositionAwareStreamReaderLegacy(stream, new EncodingOptions(), 500); - var lineCount = 0; - while (reader.ReadLine() != null) - { - lineCount += 1; - } - - Assert.That(expectedLines, Is.EqualTo(lineCount), $"Unexpected lines:\n{text}"); - } - - [Test] - [TestCase("Line 1\nLine 2\nLine 3", 3)] - [TestCase("Line 1\nLine 2\nLine 3\n", 3)] - [TestCase("Line 1\r\nLine 2\r\nLine 3", 3)] - [TestCase("Line 1\r\nLine 2\r\nLine 3\r\n", 3)] - [TestCase("Line 1\rLine 2\rLine 3", 3)] - [TestCase("Line 1\rLine 2\rLine 3\r", 3)] - public void ReadLinesWithPipelineNewLine(string text, int expectedLines) - { - using var stream = new MemoryStream(Encoding.ASCII.GetBytes(text)); - using var reader = new PositionAwareStreamReaderPipeline(stream, new EncodingOptions(), 500); - var lineCount = 0; - while (true) - { - var line = reader.ReadLine(); - if (line == null) - { - break; - } - - lineCount += 1; - - Assert.That(line.StartsWith($"Line {lineCount}", StringComparison.OrdinalIgnoreCase), $"Invalid line: {line}"); - } - - Assert.That(expectedLines, Is.EqualTo(lineCount), $"Unexpected lines:\n{text}"); - } - - [Test] - [TestCase("\n\n\n", 3)] - [TestCase("\r\n\r\n\r\n", 3)] - [TestCase("\r\r\r", 3)] - public void CountLinesWithPipelineNewLine(string text, int expectedLines) - { - using var stream = new MemoryStream(Encoding.ASCII.GetBytes(text)); - using var reader = new PositionAwareStreamReaderPipeline(stream, new EncodingOptions(), 500); - var lineCount = 0; - while (reader.ReadLine() != null) - { - lineCount += 1; - } - - Assert.That(expectedLines, Is.EqualTo(lineCount), $"Unexpected lines:\n{text}"); - } - - [Test] - public void PipelineReaderShouldTrackPositionCorrectly() - { - var text = "Line 1\nLine 2\nLine 3\n"; - using var stream = new MemoryStream(Encoding.UTF8.GetBytes(text)); - using var reader = new PositionAwareStreamReaderPipeline(stream, new EncodingOptions(), 500); - - var line1 = reader.ReadLine(); - var pos1 = reader.Position; - Assert.That(line1, Is.EqualTo("Line 1")); - Assert.That(pos1, Is.EqualTo(7)); // "Line 1\n" = 7 bytes - - var line2 = reader.ReadLine(); - var pos2 = reader.Position; - Assert.That(line2, Is.EqualTo("Line 2")); - Assert.That(pos2, Is.EqualTo(14)); // 7 + "Line 2\n" = 14 bytes - - var line3 = reader.ReadLine(); - var pos3 = reader.Position; - Assert.That(line3, Is.EqualTo("Line 3")); - Assert.That(pos3, Is.EqualTo(21)); // 14 + "Line 3\n" = 21 bytes - } - - [Test] - public void PipelineReaderShouldSupportSeeking() - { - var text = "Line 1\nLine 2\nLine 3\n"; - using var stream = new MemoryStream(Encoding.UTF8.GetBytes(text)); - using var reader = new PositionAwareStreamReaderPipeline(stream, new EncodingOptions(), 500); - - // Read first line - var line1 = reader.ReadLine(); - Assert.That(line1, Is.EqualTo("Line 1")); - - // Seek back to beginning - reader.Position = 0; - - // Should read first line again - var line1Again = reader.ReadLine(); - Assert.That(line1Again, Is.EqualTo("Line 1")); - - // Seek to middle - reader.Position = 7; // After "Line 1\n" - - // Should read second line - var line2 = reader.ReadLine(); - Assert.That(line2, Is.EqualTo("Line 2")); - } - - [Test] - public void PipelineReaderShouldHandleMaximumLineLength() - { - var longLine = new string('X', 1000); - var text = $"{longLine}\nShort line\n"; - using var stream = new MemoryStream(Encoding.UTF8.GetBytes(text)); - using var reader = new PositionAwareStreamReaderPipeline(stream, new EncodingOptions(), 100); - - // First line should be truncated to 100 chars - var line1 = reader.ReadLine(); - Assert.That(line1, Has.Length.EqualTo(100)); - Assert.That(line1, Is.EqualTo(new string('X', 100))); - - // Second line should be normal - var line2 = reader.ReadLine(); - Assert.That(line2, Is.EqualTo("Short line")); - } - - [Test] - public void PipelineReaderShouldHandleUnicode() - { - var text = "Hello 世界\nСпасибо\n"; - using var stream = new MemoryStream(Encoding.UTF8.GetBytes(text)); - using var reader = new PositionAwareStreamReaderPipeline(stream, new EncodingOptions { Encoding = Encoding.UTF8 }, 500); - - var line1 = reader.ReadLine(); - Assert.That(line1, Is.EqualTo("Hello 世界")); - - var line2 = reader.ReadLine(); - Assert.That(line2, Is.EqualTo("Спасибо")); - } - - [Test] - public void PipelineReaderShouldHandleEmptyLines() - { - var text = "Line 1\n\nLine 3\n"; - using var stream = new MemoryStream(Encoding.UTF8.GetBytes(text)); - using var reader = new PositionAwareStreamReaderPipeline(stream, new EncodingOptions(), 500); - - var line1 = reader.ReadLine(); - Assert.That(line1, Is.EqualTo("Line 1")); - - var line2 = reader.ReadLine(); - Assert.That(line2, Is.EqualTo("")); - - var line3 = reader.ReadLine(); - Assert.That(line3, Is.EqualTo("Line 3")); - - var eof = reader.ReadLine(); - Assert.That(eof, Is.Null); - } -} diff --git a/src/LogExpert.Tests/ProgressReporterTests/NullProgressReporterTests.cs b/src/LogExpert.Tests/ProgressReporterTests/NullProgressReporterTests.cs new file mode 100644 index 00000000..07a7f8b3 --- /dev/null +++ b/src/LogExpert.Tests/ProgressReporterTests/NullProgressReporterTests.cs @@ -0,0 +1,53 @@ +using LogExpert.Core.Classes.Log.ProgressReporters; + +using NUnit.Framework; + +namespace LogExpert.Tests.ProgressReporterTests; + +/// +/// Unit tests for . +/// Verifies no-op behavior and that it is safe to use in benchmarks/tests. +/// +[TestFixture] +internal sealed class NullProgressReporterTests +{ + [Test] + public void Instance_IsSingleton () + { + var a = NullProgressReporter.Instance; + var b = NullProgressReporter.Instance; + Assert.That(a, Is.SameAs(b)); + } + + [Test] + public void AllMethods_DoNotThrow () + { + var reporter = NullProgressReporter.Instance; + + Assert.DoesNotThrow(() => + { + reporter.ReportProgress("file.log", 100, 1000); + reporter.ReportComplete("file.log", 1000, 1000); + reporter.ReportNewFile("new.log", 0, 5000); + reporter.ReportLoadingStarted("file.log"); + reporter.ReportLoadingFinished(); + }); + } + + [Test] + public void Dispose_DoesNotThrow () + { + Assert.DoesNotThrow(NullProgressReporter.Instance.Dispose); + } + + [Test] + public void Dispose_CanBeCalledMultipleTimes () + { + var reporter = NullProgressReporter.Instance; + Assert.DoesNotThrow(() => + { + reporter.Dispose(); + reporter.Dispose(); + }); + } +} \ No newline at end of file diff --git a/src/LogExpert.Tests/ProgressReporterTests/PeriodicProgressReporterTests.cs b/src/LogExpert.Tests/ProgressReporterTests/PeriodicProgressReporterTests.cs new file mode 100644 index 00000000..291f7a53 --- /dev/null +++ b/src/LogExpert.Tests/ProgressReporterTests/PeriodicProgressReporterTests.cs @@ -0,0 +1,186 @@ +using LogExpert.Core.Classes.Log.ProgressReporters; +using LogExpert.Core.EventArguments; + +using NUnit.Framework; + +namespace LogExpert.Tests.ProgressReporterTests; + +/// +/// Unit tests for . +/// Uses a short dispatch interval (50ms) to keep tests fast. +/// +[TestFixture] +internal sealed class PeriodicProgressReporterTests +{ + private const int DISPATCH_MS = 50; + private const int WAIT_MS = 300; // enough for several dispatch cycles + + [Test] + public void ReportProgress_FiresLoadFileEvent () + { + using var reporter = new PeriodicProgressReporter(TimeSpan.FromMilliseconds(DISPATCH_MS)); + + LoadFileEventArgs? received = null; + reporter.LoadFile += (_, e) => received = e; + + reporter.ReportProgress("test.log", 500, 1000); + + // Wait for dispatch cycle + Thread.Sleep(WAIT_MS); + + Assert.That(received, Is.Not.Null); + Assert.That(received!.FileName, Is.EqualTo("test.log")); + Assert.That(received.ReadPos, Is.EqualTo(500)); + Assert.That(received.FileSize, Is.EqualTo(1000)); + Assert.That(received.Finished, Is.False); + Assert.That(received.NewFile, Is.False); + } + + [Test] + public void ReportComplete_FiresLoadFileEventWithFinished () + { + using var reporter = new PeriodicProgressReporter(TimeSpan.FromMilliseconds(DISPATCH_MS)); + + LoadFileEventArgs? received = null; + reporter.LoadFile += (_, e) => + { + if (e.Finished) + { + received = e; + } + }; + + reporter.ReportComplete("test.log", 1000, 1000); + Thread.Sleep(WAIT_MS); + + Assert.That(received, Is.Not.Null); + Assert.That(received!.Finished, Is.True); + } + + [Test] + public void ReportNewFile_FiresLoadFileEventWithNewFile () + { + using var reporter = new PeriodicProgressReporter(TimeSpan.FromMilliseconds(DISPATCH_MS)); + + LoadFileEventArgs? received = null; + reporter.LoadFile += (_, e) => + { + if (e.NewFile) + { + received = e; + } + }; + + reporter.ReportNewFile("new.log", 0, 5000); + Thread.Sleep(WAIT_MS); + + Assert.That(received, Is.Not.Null); + Assert.That(received!.NewFile, Is.True); + Assert.That(received.FileName, Is.EqualTo("new.log")); + } + + [Test] + public void ReportLoadingStarted_FiresLoadingStartedEvent () + { + using var reporter = new PeriodicProgressReporter(TimeSpan.FromMilliseconds(DISPATCH_MS)); + + LoadFileEventArgs? received = null; + reporter.LoadingStarted += (_, e) => received = e; + + reporter.ReportLoadingStarted("test.log"); + Thread.Sleep(WAIT_MS); + + Assert.That(received, Is.Not.Null); + Assert.That(received!.FileName, Is.EqualTo("test.log")); + } + + [Test] + public void ReportLoadingFinished_FiresLoadingFinishedEvent () + { + using var reporter = new PeriodicProgressReporter(TimeSpan.FromMilliseconds(DISPATCH_MS)); + + var fired = false; + reporter.LoadingFinished += (_, _) => fired = true; + + reporter.ReportLoadingFinished(); + Thread.Sleep(WAIT_MS); + + Assert.That(fired, Is.True); + } + + [Test] + public void MultipleProgressReports_CoalescedToLatest () + { + using var reporter = new PeriodicProgressReporter(TimeSpan.FromMilliseconds(DISPATCH_MS)); + + LoadFileEventArgs? received = null; + reporter.LoadFile += (_, e) => + { + if (!e.Finished) + { + received = e; + } + }; + + // Fire many progress reports rapidly — only latest should be dispatched + for (var i = 0; i < 100; i++) + { + reporter.ReportProgress("test.log", i * 100, 10000); + } + + Thread.Sleep(WAIT_MS); + + Assert.That(received, Is.Not.Null); + // The dispatched position should be the latest (or near-latest) value + Assert.That(received!.ReadPos, Is.GreaterThanOrEqualTo(9000)); + } + + [Test] + public void DispatchOrder_StartedBeforeProgress () + { + using var reporter = new PeriodicProgressReporter(TimeSpan.FromMilliseconds(DISPATCH_MS)); + + var order = new List(); + reporter.LoadingStarted += (_, _) => order.Add("started"); + reporter.LoadFile += (_, _) => order.Add("progress"); + + reporter.ReportLoadingStarted("test.log"); + reporter.ReportProgress("test.log", 500, 1000); + Thread.Sleep(WAIT_MS); + + Assert.That(order, Has.Count.GreaterThanOrEqualTo(2)); + var startedIdx = order.IndexOf("started"); + var progressIdx = order.IndexOf("progress"); + Assert.That(startedIdx, Is.GreaterThanOrEqualTo(0), "LoadingStarted not fired"); + Assert.That(progressIdx, Is.GreaterThanOrEqualTo(0), "LoadFile not fired"); + Assert.That(startedIdx, Is.LessThan(progressIdx), "Started should fire before progress"); + } + + [Test] + public void Dispose_StopsDispatchLoop () + { + var reporter = new PeriodicProgressReporter(TimeSpan.FromMilliseconds(DISPATCH_MS)); + reporter.Dispose(); + + // After dispose, reports should not throw but events won't fire + Assert.DoesNotThrow(() => reporter.ReportProgress("test.log", 100, 1000)); + } + + [Test] + public void NoSubscribers_DoesNotThrow () + { + using var reporter = new PeriodicProgressReporter(TimeSpan.FromMilliseconds(DISPATCH_MS)); + + // Report all event types with no subscribers — should not throw + Assert.DoesNotThrow(() => + { + reporter.ReportLoadingStarted("test.log"); + reporter.ReportProgress("test.log", 100, 1000); + reporter.ReportNewFile("new.log", 0, 5000); + reporter.ReportComplete("test.log", 1000, 1000); + reporter.ReportLoadingFinished(); + }); + + Thread.Sleep(WAIT_MS); + } +} \ No newline at end of file diff --git a/src/LogExpert.Tests/ReaderTest.cs b/src/LogExpert.Tests/ReaderTest.cs index 791555bd..8c71d36e 100644 --- a/src/LogExpert.Tests/ReaderTest.cs +++ b/src/LogExpert.Tests/ReaderTest.cs @@ -1,6 +1,6 @@ using System.Text; -using LogExpert.Core.Classes.Log; +using LogExpert.Core.Classes.Log.Streamreaders; using LogExpert.Core.Entities; using LogExpert.Core.Interfaces; diff --git a/src/LogExpert.Tests/Services/FileOperationServiceTests.cs b/src/LogExpert.Tests/Services/FileOperationServiceTests.cs index 114a0c83..39b72151 100644 --- a/src/LogExpert.Tests/Services/FileOperationServiceTests.cs +++ b/src/LogExpert.Tests/Services/FileOperationServiceTests.cs @@ -37,6 +37,13 @@ internal class FileOperationServiceTests : IDisposable private bool _disposed; + [OneTimeSetUp] + public void OneTimeSetUp () + { + var dir = Path.GetDirectoryName(typeof(FileOperationServiceTests).Assembly.Location)!; + _ = PluginRegistry.PluginRegistry.Create(dir, 500); + } + [SetUp] public void Setup () { diff --git a/src/LogExpert.Tests/Services/TabControllerTests.cs b/src/LogExpert.Tests/Services/TabControllerTests.cs index 8c64d8ec..bcd23ac9 100644 --- a/src/LogExpert.Tests/Services/TabControllerTests.cs +++ b/src/LogExpert.Tests/Services/TabControllerTests.cs @@ -1,5 +1,4 @@ using System.Runtime.Versioning; -using System.Windows.Forms; using LogExpert.UI.Controls.LogWindow; using LogExpert.UI.Services.TabControllerService; @@ -12,13 +11,11 @@ namespace LogExpert.Tests.Services; /// /// Unit tests for TabController. -/// -/// Note: Many tests are limited because LogWindow is a complex WinForms control -/// that cannot be easily mocked or subclassed. Tests that require actual LogWindow -/// instances would need to be run as integration tests with full UI infrastructure. -/// -/// These tests focus on the core TabController functionality that can be tested -/// without instantiating LogWindow objects. +/// Note: Many tests are limited because LogWindow is a complex WinForms control that cannot be easily mocked or +/// subclassed. Tests that require actual LogWindow instances would need to be run as integration tests with full UI +/// infrastructure. +/// These tests focus on the core TabController functionality that can be tested without instantiating LogWindow +/// objects. /// [TestFixture] [SupportedOSPlatform("windows")] @@ -31,7 +28,7 @@ internal class TabControllerTests : IDisposable private bool _disposed; [SetUp] - public void Setup() + public void Setup () { // Create a real Form and DockPanel for testing // This is necessary because DockPanel requires WinForms infrastructure @@ -48,20 +45,22 @@ public void Setup() } [TearDown] - public void TearDown() + public void TearDown () { _tabController?.Dispose(); _testForm?.Close(); _testForm?.Dispose(); } - public void Dispose() + #region IDisposable Implementation + + public void Dispose () { Dispose(true); GC.SuppressFinalize(this); } - protected virtual void Dispose(bool disposing) + protected virtual void Dispose (bool disposing) { if (_disposed) { @@ -77,10 +76,12 @@ protected virtual void Dispose(bool disposing) _disposed = true; } + #endregion + #region Constructor Tests [Test] - public void Constructor_WithDockPanel_InitializesSuccessfully() + public void Constructor_WithDockPanel_InitializesSuccessfully () { // Arrange & Act - already done in Setup @@ -90,14 +91,14 @@ public void Constructor_WithDockPanel_InitializesSuccessfully() } [Test] - public void Constructor_WithNullDockPanel_ThrowsArgumentNullException() + public void Constructor_WithNullDockPanel_ThrowsArgumentNullException () { // Arrange & Act & Assert _ = Assert.Throws(() => new TabController(null)); } [Test] - public void Constructor_WithoutDockPanel_CreatesUninitializedController() + public void Constructor_WithoutDockPanel_CreatesUninitializedController () { // Arrange & Act using var controller = new TabController(); @@ -112,7 +113,7 @@ public void Constructor_WithoutDockPanel_CreatesUninitializedController() #region InitializeDockPanel Tests [Test] - public void InitializeDockPanel_WithValidDockPanel_Succeeds() + public void InitializeDockPanel_WithValidDockPanel_Succeeds () { // Arrange using var controller = new TabController(); @@ -125,7 +126,7 @@ public void InitializeDockPanel_WithValidDockPanel_Succeeds() } [Test] - public void InitializeDockPanel_WithNullDockPanel_ThrowsArgumentNullException() + public void InitializeDockPanel_WithNullDockPanel_ThrowsArgumentNullException () { // Arrange using var controller = new TabController(); @@ -135,7 +136,7 @@ public void InitializeDockPanel_WithNullDockPanel_ThrowsArgumentNullException() } [Test] - public void InitializeDockPanel_WhenAlreadyInitialized_ThrowsInvalidOperationException() + public void InitializeDockPanel_WhenAlreadyInitialized_ThrowsInvalidOperationException () { // Arrange using var controller = new TabController(_dockPanel); @@ -154,7 +155,7 @@ public void InitializeDockPanel_WhenAlreadyInitialized_ThrowsInvalidOperationExc #region GetAllWindowsFromDockPanel Tests [Test] - public void GetAllWindowsFromDockPanel_WhenNotInitialized_ReturnsEmptyList() + public void GetAllWindowsFromDockPanel_WhenNotInitialized_ReturnsEmptyList () { // Arrange using var controller = new TabController(); @@ -167,7 +168,7 @@ public void GetAllWindowsFromDockPanel_WhenNotInitialized_ReturnsEmptyList() } [Test] - public void GetAllWindowsFromDockPanel_WhenInitializedButEmpty_ReturnsEmptyList() + public void GetAllWindowsFromDockPanel_WhenInitializedButEmpty_ReturnsEmptyList () { // Arrange - already done in Setup @@ -179,7 +180,7 @@ public void GetAllWindowsFromDockPanel_WhenInitializedButEmpty_ReturnsEmptyList( } [Test] - public void GetAllWindowsFromDockPanel_ReturnsReadOnlyList() + public void GetAllWindowsFromDockPanel_ReturnsReadOnlyList () { // Arrange - already done in Setup @@ -195,7 +196,7 @@ public void GetAllWindowsFromDockPanel_ReturnsReadOnlyList() #region GetAllWindows Tests [Test] - public void GetAllWindows_WhenEmpty_ReturnsEmptyList() + public void GetAllWindows_WhenEmpty_ReturnsEmptyList () { // Arrange - already done in Setup @@ -212,7 +213,7 @@ public void GetAllWindows_WhenEmpty_ReturnsEmptyList() #region GetWindowCount Tests [Test] - public void GetWindowCount_WhenEmpty_ReturnsZero() + public void GetWindowCount_WhenEmpty_ReturnsZero () { // Arrange - already done in Setup @@ -228,7 +229,7 @@ public void GetWindowCount_WhenEmpty_ReturnsZero() #region HasWindow Tests [Test] - public void HasWindow_WithNullWindow_ReturnsFalse() + public void HasWindow_WithNullWindow_ReturnsFalse () { // Arrange - already done in Setup @@ -244,7 +245,7 @@ public void HasWindow_WithNullWindow_ReturnsFalse() #region GetActiveWindow Tests [Test] - public void GetActiveWindow_WhenNoWindowActive_ReturnsNull() + public void GetActiveWindow_WhenNoWindowActive_ReturnsNull () { // Arrange - already done in Setup @@ -260,7 +261,7 @@ public void GetActiveWindow_WhenNoWindowActive_ReturnsNull() #region FindWindowByFileName Tests [Test] - public void FindWindowByFileName_WithNullFileName_ReturnsNull() + public void FindWindowByFileName_WithNullFileName_ReturnsNull () { // Arrange - already done in Setup @@ -272,7 +273,7 @@ public void FindWindowByFileName_WithNullFileName_ReturnsNull() } [Test] - public void FindWindowByFileName_WithEmptyFileName_ReturnsNull() + public void FindWindowByFileName_WithEmptyFileName_ReturnsNull () { // Arrange - already done in Setup @@ -284,7 +285,7 @@ public void FindWindowByFileName_WithEmptyFileName_ReturnsNull() } [Test] - public void FindWindowByFileName_WhenNoWindowsExist_ReturnsNull() + public void FindWindowByFileName_WhenNoWindowsExist_ReturnsNull () { // Arrange - already done in Setup @@ -300,7 +301,7 @@ public void FindWindowByFileName_WhenNoWindowsExist_ReturnsNull() #region AddWindow Tests [Test] - public void AddWindow_WithNullWindow_ThrowsArgumentNullException() + public void AddWindow_WithNullWindow_ThrowsArgumentNullException () { // Arrange - already done in Setup @@ -309,7 +310,7 @@ public void AddWindow_WithNullWindow_ThrowsArgumentNullException() } [Test] - public void AddWindow_WhenNotInitialized_ThrowsInvalidOperationException() + public void AddWindow_WhenNotInitialized_ThrowsInvalidOperationException () { // Arrange using var controller = new TabController(); @@ -327,7 +328,7 @@ public void AddWindow_WhenNotInitialized_ThrowsInvalidOperationException() #region RemoveWindow Tests [Test] - public void RemoveWindow_WithNullWindow_DoesNotThrow() + public void RemoveWindow_WithNullWindow_DoesNotThrow () { // Arrange - already done in Setup @@ -340,7 +341,7 @@ public void RemoveWindow_WithNullWindow_DoesNotThrow() #region CloseWindow Tests [Test] - public void CloseWindow_WithNullWindow_DoesNotThrow() + public void CloseWindow_WithNullWindow_DoesNotThrow () { // Arrange - already done in Setup @@ -353,7 +354,7 @@ public void CloseWindow_WithNullWindow_DoesNotThrow() #region CloseAllWindows Tests [Test] - public void CloseAllWindows_WhenEmpty_DoesNotThrow() + public void CloseAllWindows_WhenEmpty_DoesNotThrow () { // Arrange - already done in Setup @@ -366,7 +367,7 @@ public void CloseAllWindows_WhenEmpty_DoesNotThrow() #region CloseAllExcept Tests [Test] - public void CloseAllExcept_WithNullWindow_DoesNotThrow() + public void CloseAllExcept_WithNullWindow_DoesNotThrow () { // Arrange - already done in Setup @@ -379,7 +380,7 @@ public void CloseAllExcept_WithNullWindow_DoesNotThrow() #region ActivateWindow Tests [Test] - public void ActivateWindow_WithNullWindow_DoesNotThrow() + public void ActivateWindow_WithNullWindow_DoesNotThrow () { // Arrange - already done in Setup @@ -392,7 +393,7 @@ public void ActivateWindow_WithNullWindow_DoesNotThrow() #region SwitchToNextWindow Tests [Test] - public void SwitchToNextWindow_WhenEmpty_DoesNotThrow() + public void SwitchToNextWindow_WhenEmpty_DoesNotThrow () { // Arrange - already done in Setup @@ -405,7 +406,7 @@ public void SwitchToNextWindow_WhenEmpty_DoesNotThrow() #region SwitchToPreviousWindow Tests [Test] - public void SwitchToPreviousWindow_WhenEmpty_DoesNotThrow() + public void SwitchToPreviousWindow_WhenEmpty_DoesNotThrow () { // Arrange - already done in Setup @@ -418,7 +419,7 @@ public void SwitchToPreviousWindow_WhenEmpty_DoesNotThrow() #region Dispose Tests [Test] - public void Dispose_MultipleCallsDoNotThrow() + public void Dispose_MultipleCallsDoNotThrow () { // Arrange using var controller = new TabController(_dockPanel); diff --git a/src/LogExpert.Tests/StreamReaderTests/LogStreamReaderTest.cs b/src/LogExpert.Tests/StreamReaderTests/LogStreamReaderTest.cs new file mode 100644 index 00000000..d1203189 --- /dev/null +++ b/src/LogExpert.Tests/StreamReaderTests/LogStreamReaderTest.cs @@ -0,0 +1,422 @@ +using System.Text; + +using LogExpert.Core.Classes.Log.Streamreaders; +using LogExpert.Core.Entities; + +using NUnit.Framework; + +namespace LogExpert.Tests.StreamReaderTests; + +[TestFixture] +public class LogStreamReaderTest +{ + [Test] + [TestCase("Line 1\nLine 2\nLine 3", 3)] + [TestCase("Line 1\nLine 2\nLine 3\n", 3)] + [TestCase("Line 1\r\nLine 2\r\nLine 3", 3)] + [TestCase("Line 1\r\nLine 2\r\nLine 3\r\n", 3)] + [TestCase("Line 1\rLine 2\rLine 3", 3)] + [TestCase("Line 1\rLine 2\rLine 3\r", 3)] + public void ReadLinesWithSystemNewLine (string text, int expectedLines) + { + using var stream = new MemoryStream(Encoding.ASCII.GetBytes(text)); + using var reader = new PositionAwareStreamReaderSystem(stream, new EncodingOptions(), 500); + var lineCount = 0; + while (true) + { + var line = reader.ReadLine(); + if (line == null) + { + break; + } + + lineCount += 1; + + Assert.That(line.StartsWith($"Line {lineCount}", StringComparison.OrdinalIgnoreCase), $"Invalid line: {line}"); + } + + Assert.That(expectedLines, Is.EqualTo(lineCount), $"Unexpected lines:\n{text}"); + } + + [Test] + [TestCase("\n\n\n", 3)] + [TestCase("\r\n\r\n\r\n", 3)] + [TestCase("\r\r\r", 3)] + public void CountLinesWithSystemNewLine (string text, int expectedLines) + { + using var stream = new MemoryStream(Encoding.ASCII.GetBytes(text)); + using var reader = new PositionAwareStreamReaderSystem(stream, new EncodingOptions(), 500); + var lineCount = 0; + while (reader.ReadLine() != null) + { + lineCount += 1; + } + + Assert.That(expectedLines, Is.EqualTo(lineCount), $"Unexpected lines:\n{text}"); + } + + [Test] + [TestCase("Line 1\nLine 2\nLine 3", 3)] + [TestCase("Line 1\nLine 2\nLine 3\n", 3)] + [TestCase("Line 1\r\nLine 2\r\nLine 3", 3)] + [TestCase("Line 1\r\nLine 2\r\nLine 3\r\n", 3)] + [TestCase("Line 1\rLine 2\rLine 3", 3)] + [TestCase("Line 1\rLine 2\rLine 3\r", 3)] + public void ReadLinesWithLegacyNewLine (string text, int expectedLines) + { + using var stream = new MemoryStream(Encoding.ASCII.GetBytes(text)); + using var reader = new PositionAwareStreamReaderLegacy(stream, new EncodingOptions(), 500); + var lineCount = 0; + while (true) + { + var line = reader.ReadLine(); + if (line == null) + { + break; + } + + lineCount += 1; + + Assert.That(line.StartsWith($"Line {lineCount}", StringComparison.OrdinalIgnoreCase), $"Invalid line: {line}"); + } + + Assert.That(expectedLines, Is.EqualTo(lineCount), $"Unexpected lines:\n{text}"); + } + [Test] + [TestCase("\n\n\n", 3)] + [TestCase("\r\n\r\n\r\n", 3)] + [TestCase("\r\r\r", 3)] + public void CountLinesWithLegacyNewLine (string text, int expectedLines) + { + using var stream = new MemoryStream(Encoding.ASCII.GetBytes(text)); + using var reader = new PositionAwareStreamReaderLegacy(stream, new EncodingOptions(), 500); + var lineCount = 0; + while (reader.ReadLine() != null) + { + lineCount += 1; + } + + Assert.That(expectedLines, Is.EqualTo(lineCount), $"Unexpected lines:\n{text}"); + } + + [Test] + [TestCase("Line 1\nLine 2\nLine 3", 3)] + [TestCase("Line 1\nLine 2\nLine 3\n", 3)] + [TestCase("Line 1\r\nLine 2\r\nLine 3", 3)] + [TestCase("Line 1\r\nLine 2\r\nLine 3\r\n", 3)] + [TestCase("Line 1\rLine 2\rLine 3", 3)] + [TestCase("Line 1\rLine 2\rLine 3\r", 3)] + public void ReadLinesWithPipelineNewLine (string text, int expectedLines) + { + using var stream = new MemoryStream(Encoding.ASCII.GetBytes(text)); + using var reader = new PositionAwareStreamReaderPipeline(stream, new EncodingOptions(), 500); + var lineCount = 0; + while (true) + { + var line = reader.ReadLine(); + if (line == null) + { + break; + } + + lineCount += 1; + + Assert.That(line.StartsWith($"Line {lineCount}", StringComparison.OrdinalIgnoreCase), $"Invalid line: {line}"); + } + + Assert.That(expectedLines, Is.EqualTo(lineCount), $"Unexpected lines:\n{text}"); + } + + [Test] + [TestCase("\n\n\n", 3)] + [TestCase("\r\n\r\n\r\n", 3)] + [TestCase("\r\r\r", 3)] + public void CountLinesWithPipelineNewLine (string text, int expectedLines) + { + using var stream = new MemoryStream(Encoding.ASCII.GetBytes(text)); + using var reader = new PositionAwareStreamReaderPipeline(stream, new EncodingOptions(), 500); + var lineCount = 0; + while (reader.ReadLine() != null) + { + lineCount += 1; + } + + Assert.That(expectedLines, Is.EqualTo(lineCount), $"Unexpected lines:\n{text}"); + } + + [Test] + public void PipelineReaderShouldTrackPositionCorrectly () + { + var text = "Line 1\nLine 2\nLine 3\n"; + using var stream = new MemoryStream(Encoding.UTF8.GetBytes(text)); + using var reader = new PositionAwareStreamReaderPipeline(stream, new EncodingOptions(), 500); + + var line1 = reader.ReadLine(); + var pos1 = reader.Position; + Assert.That(line1, Is.EqualTo("Line 1")); + Assert.That(pos1, Is.EqualTo(7)); // "Line 1\n" = 7 bytes + + var line2 = reader.ReadLine(); + var pos2 = reader.Position; + Assert.That(line2, Is.EqualTo("Line 2")); + Assert.That(pos2, Is.EqualTo(14)); // 7 + "Line 2\n" = 14 bytes + + var line3 = reader.ReadLine(); + var pos3 = reader.Position; + Assert.That(line3, Is.EqualTo("Line 3")); + Assert.That(pos3, Is.EqualTo(21)); // 14 + "Line 3\n" = 21 bytes + } + + [Test] + public void PipelineReaderShouldSupportSeeking () + { + var text = "Line 1\nLine 2\nLine 3\n"; + using var stream = new MemoryStream(Encoding.UTF8.GetBytes(text)); + using var reader = new PositionAwareStreamReaderPipeline(stream, new EncodingOptions(), 500); + + // Read first line + var line1 = reader.ReadLine(); + Assert.That(line1, Is.EqualTo("Line 1")); + + // Seek back to beginning + reader.Position = 0; + + // Should read first line again + var line1Again = reader.ReadLine(); + Assert.That(line1Again, Is.EqualTo("Line 1")); + + // Seek to middle + reader.Position = 7; // After "Line 1\n" + + // Should read second line + var line2 = reader.ReadLine(); + Assert.That(line2, Is.EqualTo("Line 2")); + } + + [Test] + public void PipelineReaderShouldHandleMaximumLineLength () + { + var longLine = new string('X', 1000); + var text = $"{longLine}\nShort line\n"; + using var stream = new MemoryStream(Encoding.UTF8.GetBytes(text)); + using var reader = new PositionAwareStreamReaderPipeline(stream, new EncodingOptions(), 100); + + // First line should be truncated to 100 chars + var line1 = reader.ReadLine(); + Assert.That(line1, Has.Length.EqualTo(100)); + Assert.That(line1, Is.EqualTo(new string('X', 100))); + + // Second line should be normal + var line2 = reader.ReadLine(); + Assert.That(line2, Is.EqualTo("Short line")); + } + + [Test] + public void PipelineReaderShouldHandleUnicode () + { + var text = "Hello 世界\nСпасибо\n"; + using var stream = new MemoryStream(Encoding.UTF8.GetBytes(text)); + using var reader = new PositionAwareStreamReaderPipeline(stream, new EncodingOptions { Encoding = Encoding.UTF8 }, 500); + + var line1 = reader.ReadLine(); + Assert.That(line1, Is.EqualTo("Hello 世界")); + + var line2 = reader.ReadLine(); + Assert.That(line2, Is.EqualTo("Спасибо")); + } + + [Test] + public void PipelineReaderShouldHandleEmptyLines () + { + var text = "Line 1\n\nLine 3\n"; + using var stream = new MemoryStream(Encoding.UTF8.GetBytes(text)); + using var reader = new PositionAwareStreamReaderPipeline(stream, new EncodingOptions(), 500); + + var line1 = reader.ReadLine(); + Assert.That(line1, Is.EqualTo("Line 1")); + + var line2 = reader.ReadLine(); + Assert.That(line2, Is.EqualTo("")); + + var line3 = reader.ReadLine(); + Assert.That(line3, Is.EqualTo("Line 3")); + + var eof = reader.ReadLine(); + Assert.That(eof, Is.Null); + } + + [Test] + [TestCase("Line 1\nLine 2\nLine 3", 3)] + [TestCase("Line 1\nLine 2\nLine 3\n", 3)] + [TestCase("Line 1\r\nLine 2\r\nLine 3", 3)] + [TestCase("Line 1\r\nLine 2\r\nLine 3\r\n", 3)] + [TestCase("Line 1\rLine 2\rLine 3", 3)] + [TestCase("Line 1\rLine 2\rLine 3\r", 3)] + public void TryReadLine_ReturnsCorrectContent_SystemReader (string text, int expectedLines) + { + using var stream = new MemoryStream(Encoding.ASCII.GetBytes(text)); + using var reader = new PositionAwareStreamReaderSystem(stream, new EncodingOptions(), 500); + var lineCount = 0; + while (reader.TryReadLine(out var lineMemory)) + { + lineCount++; + var line = lineMemory.Span.ToString(); + Assert.That(line.StartsWith($"Line {lineCount}", StringComparison.OrdinalIgnoreCase), $"Invalid line: {line}"); + } + + Assert.That(lineCount, Is.EqualTo(expectedLines)); + } + + [Test] + [TestCase("\n\n\n", 3)] + [TestCase("\r\n\r\n\r\n", 3)] + [TestCase("\r\r\r", 3)] + public void TryReadLine_CountsEmptyLines_SystemReader (string text, int expectedLines) + { + using var stream = new MemoryStream(Encoding.ASCII.GetBytes(text)); + using var reader = new PositionAwareStreamReaderSystem(stream, new EncodingOptions(), 500); + var lineCount = 0; + while (reader.TryReadLine(out _)) + { + lineCount++; + } + + Assert.That(lineCount, Is.EqualTo(expectedLines)); + } + + [Test] + public void TryReadLine_TracksPositionCorrectly_SystemReader () + { + var text = "Line 1\nLine 2\nLine 3\n"; + using var stream = new MemoryStream(Encoding.UTF8.GetBytes(text)); + using var reader = new PositionAwareStreamReaderSystem(stream, new EncodingOptions(), 500); + + Assert.That(reader.TryReadLine(out var line1), Is.True); + Assert.That(line1.Span.ToString(), Is.EqualTo("Line 1")); + Assert.That(reader.Position, Is.EqualTo(7)); // "Line 1\n" = 7 bytes + + Assert.That(reader.TryReadLine(out var line2), Is.True); + Assert.That(line2.Span.ToString(), Is.EqualTo("Line 2")); + Assert.That(reader.Position, Is.EqualTo(14)); + + Assert.That(reader.TryReadLine(out var line3), Is.True); + Assert.That(line3.Span.ToString(), Is.EqualTo("Line 3")); + Assert.That(reader.Position, Is.EqualTo(21)); + + Assert.That(reader.TryReadLine(out _), Is.False); // EOF + } + + [Test] + public void TryReadLine_ReturnsBlockBackedMemory_NotStringBacked () + { + var text = "Hello World\nSecond Line\n"; + using var stream = new MemoryStream(Encoding.UTF8.GetBytes(text)); + using var reader = new PositionAwareStreamReaderSystem(stream, new EncodingOptions(), 500); + + Assert.That(reader.TryReadLine(out var lineMemory), Is.True); + + // Verify the memory is backed by a char[] from the block allocator, not a string. + // MemoryMarshal.TryGetArray succeeds for array-backed Memory but fails for string-backed Memory. + var success = System.Runtime.InteropServices.MemoryMarshal.TryGetArray(lineMemory, out var segment); + Assert.That(success, Is.True, "Memory should be backed by a char[] (block-allocated), not a string"); + Assert.That(segment.Array, Is.Not.Null); + } + + [Test] + public void TryReadLine_BlockAllocatorHasBlocks_AfterReading () + { + var text = "Line 1\nLine 2\nLine 3\n"; + using var stream = new MemoryStream(Encoding.UTF8.GetBytes(text)); + using var reader = new PositionAwareStreamReaderSystem(stream, new EncodingOptions(), 500); + + while (reader.TryReadLine(out _)) + { + // Intentionally empty: consume all lines to advance reader state. + } + + // The allocator should have at least 1 block + Assert.That(reader.BlockAllocator.BlockCount, Is.GreaterThanOrEqualTo(1)); + } + + [Test] + public void TryReadLine_DetachBlocks_TransfersOwnership () + { + var text = "Line 1\nLine 2\nLine 3\n"; + using var stream = new MemoryStream(Encoding.UTF8.GetBytes(text)); + using var reader = new PositionAwareStreamReaderSystem(stream, new EncodingOptions(), 500); + + while (reader.TryReadLine(out _)) + { + // Intentionally empty: consume all lines to advance reader state. + } + + var blocks = reader.BlockAllocator.DetachBlocks(); + Assert.That(blocks, Has.Count.GreaterThanOrEqualTo(1)); + + // After detach, allocator should have a fresh block + Assert.That(reader.BlockAllocator.BlockCount, Is.EqualTo(1)); + } + + [Test] + public void TryReadLine_MatchesReadLine_ContentAndPosition () + { + // Verify TryReadLine produces identical content and position tracking as ReadLine + var text = "2026-04-23 12:00:00 [INFO] Thread-1 SomeClass - Message 1\n" + + "2026-04-23 12:00:01 [WARN] Thread-2 OtherClass - Message 2\n" + + "Short\n" + + "\n" + // empty line + "Last line"; + + using var stream1 = new MemoryStream(Encoding.UTF8.GetBytes(text)); + using var stream2 = new MemoryStream(Encoding.UTF8.GetBytes(text)); + using var readLineReader = new PositionAwareStreamReaderSystem(stream1, new EncodingOptions(), 500); + using var tryReadLineReader = new PositionAwareStreamReaderSystem(stream2, new EncodingOptions(), 500); + + while (true) + { + var stringLine = readLineReader.ReadLine(); + var tryResult = tryReadLineReader.TryReadLine(out var memoryLine); + + if (stringLine == null) + { + Assert.That(tryResult, Is.False, "TryReadLine should return false at EOF when ReadLine returns null"); + break; + } + + Assert.That(tryResult, Is.True); + Assert.That(memoryLine.Span.ToString(), Is.EqualTo(stringLine), "Content mismatch between ReadLine and TryReadLine"); + Assert.That(tryReadLineReader.Position, Is.EqualTo(readLineReader.Position), "Position mismatch between ReadLine and TryReadLine"); + } + } + + [Test] + public void TryReadLine_RespectsMaximumLineLength () + { + var longLine = new string('X', 1000); + var text = $"{longLine}\nShort\n"; + using var stream = new MemoryStream(Encoding.UTF8.GetBytes(text)); + using var reader = new PositionAwareStreamReaderSystem(stream, new EncodingOptions(), 500); + + Assert.That(reader.TryReadLine(out var lineMemory), Is.True); + Assert.That(lineMemory.Length, Is.EqualTo(500), "Line should be truncated to MaximumLineLength"); + + Assert.That(reader.TryReadLine(out var line2), Is.True); + Assert.That(line2.Span.ToString(), Is.EqualTo("Short")); + } + + [Test] + public void TryReadLine_UTF8_MultiByteCharacters () + { + // Japanese characters: 3 bytes each in UTF-8 + var text = "日本語テスト\nLine 2\n"; + using var stream = new MemoryStream(Encoding.UTF8.GetBytes(text)); + using var reader = new PositionAwareStreamReaderSystem(stream, new EncodingOptions { Encoding = Encoding.UTF8 }, 500); + + Assert.That(reader.TryReadLine(out var line1), Is.True); + Assert.That(line1.Span.ToString(), Is.EqualTo("日本語テスト")); + Assert.That(reader.Position, Is.EqualTo(Encoding.UTF8.GetByteCount("日本語テスト\n"))); + + Assert.That(reader.TryReadLine(out var line2), Is.True); + Assert.That(line2.Span.ToString(), Is.EqualTo("Line 2")); + } +} diff --git a/src/LogExpert.Tests/StreamReaderTests/LogfileReaderBlockAllocationTests.cs b/src/LogExpert.Tests/StreamReaderTests/LogfileReaderBlockAllocationTests.cs new file mode 100644 index 00000000..6bd1ae97 --- /dev/null +++ b/src/LogExpert.Tests/StreamReaderTests/LogfileReaderBlockAllocationTests.cs @@ -0,0 +1,252 @@ +using System.Text; + +using LogExpert.Core.Classes.Log; +using LogExpert.Core.Entities; +using LogExpert.Core.Enums; + +using NUnit.Framework; + +namespace LogExpert.Tests.StreamReaderTests; + +[TestFixture] +public class LogfileReaderBlockAllocationTests +{ + private string _tempFile; + + [SetUp] + public void Setup () + { + _tempFile = Path.GetTempFileName(); + _ = PluginRegistry.PluginRegistry.Create(Path.GetDirectoryName(_tempFile)!, 500); + } + + [TearDown] + public void Cleanup () + { + if (File.Exists(_tempFile)) + { + File.Delete(_tempFile); + } + } + + [Test] + [TestCase(10)] + [TestCase(100)] + [TestCase(1_000)] + [TestCase(10_000)] + public void ReadFiles_AllLinesCorrect_WithBlockAllocation (int lineCount) + { + GenerateLogFile(_tempFile, lineCount); + + using var reader = new LogfileReader( + _tempFile, + new EncodingOptions { Encoding = Encoding.UTF8 }, + multiFile: false, + bufferCount: 100, + linesPerBuffer: 500, + new MultiFileOptions(), + ReaderType.System, + PluginRegistry.PluginRegistry.Instance, + maximumLineLength: 500, + progressReporter: Core.Classes.Log.ProgressReporters.NullProgressReporter.Instance); + + reader.ReadFiles(); + + Assert.That(reader.LineCount, Is.EqualTo(lineCount)); + + // Verify first, middle, and last lines + VerifyLine(reader, 0, 0); + VerifyLine(reader, lineCount / 2, lineCount / 2); + VerifyLine(reader, lineCount - 1, lineCount - 1); + } + + [Test] + [TestCase(10)] + [TestCase(100)] + [TestCase(1_000)] + [TestCase(10_000)] + public void ReadFiles_AllLinesCorrect_WithDirectRead (int lineCount) + { + GenerateLogFile(_tempFile, lineCount); + + using var reader = new LogfileReader( + _tempFile, + new EncodingOptions { Encoding = Encoding.UTF8 }, + multiFile: false, + bufferCount: 100, + linesPerBuffer: 500, + new MultiFileOptions(), + ReaderType.SystemDirect, + PluginRegistry.PluginRegistry.Instance, + maximumLineLength: 500, + progressReporter: Core.Classes.Log.ProgressReporters.NullProgressReporter.Instance); + + reader.ReadFiles(); + + Assert.That(reader.LineCount, Is.EqualTo(lineCount)); + + // Verify first, middle, and last lines + VerifyLine(reader, 0, 0); + VerifyLine(reader, lineCount / 2, lineCount / 2); + VerifyLine(reader, lineCount - 1, lineCount - 1); + } + + [Test] + public void ReadFiles_EmptyLines_PreservedCorrectly () + { + File.WriteAllText(_tempFile, "Line 1\n\nLine 3\n\nLine 5\n", Encoding.UTF8); + + using var reader = new LogfileReader( + _tempFile, + new EncodingOptions { Encoding = Encoding.UTF8 }, + multiFile: false, + bufferCount: 100, + linesPerBuffer: 500, + new MultiFileOptions(), + ReaderType.System, + PluginRegistry.PluginRegistry.Instance, + maximumLineLength: 500, + progressReporter: Core.Classes.Log.ProgressReporters.NullProgressReporter.Instance); + + reader.ReadFiles(); + + Assert.That(reader.LineCount, Is.EqualTo(5)); + + var line1 = reader.GetLogLineMemory(0); + var line2 = reader.GetLogLineMemory(1); + var line3 = reader.GetLogLineMemory(2); + + Assert.That(line1?.FullLine.Span.ToString(), Is.EqualTo("Line 1")); + Assert.That(line2?.FullLine.IsEmpty, Is.True); // empty line + Assert.That(line3?.FullLine.Span.ToString(), Is.EqualTo("Line 3")); + } + + [Test] + public void ReadFiles_UTF8MultiByte_ContentPreserved () + { + File.WriteAllText(_tempFile, "日本語テスト\nÄÖÜ äöü\nLine 3\n", Encoding.UTF8); + + using var reader = new LogfileReader( + _tempFile, + new EncodingOptions { Encoding = Encoding.UTF8 }, + multiFile: false, + bufferCount: 100, + linesPerBuffer: 500, + new MultiFileOptions(), + ReaderType.System, + PluginRegistry.PluginRegistry.Instance, + maximumLineLength: 500, + progressReporter: Core.Classes.Log.ProgressReporters.NullProgressReporter.Instance); + + reader.ReadFiles(); + + Assert.That(reader.LineCount, Is.EqualTo(3)); + + var line1 = reader.GetLogLineMemory(0); + var line2 = reader.GetLogLineMemory(1); + + Assert.That(line1?.FullLine.Span.ToString(), Is.EqualTo("日本語テスト")); + Assert.That(line2?.FullLine.Span.ToString(), Is.EqualTo("ÄÖÜ äöü")); + } + + [Test] + public void ReadFiles_LongLine_TruncatedToMaxLength () + { + var longLine = new string('X', 1000); + File.WriteAllText(_tempFile, $"{longLine}\nShort\n", Encoding.UTF8); + + using var reader = new LogfileReader( + _tempFile, + new EncodingOptions { Encoding = Encoding.UTF8 }, + multiFile: false, + bufferCount: 100, + linesPerBuffer: 500, + new MultiFileOptions(), + ReaderType.System, + PluginRegistry.PluginRegistry.Instance, + maximumLineLength: 500, + progressReporter: Core.Classes.Log.ProgressReporters.NullProgressReporter.Instance); + + reader.ReadFiles(); + + var line1 = reader.GetLogLineMemory(0); + Assert.That(line1?.FullLine.Length, Is.EqualTo(500)); + } + + [Test] + public void ReadFiles_LineContentBackedByPooledMemory () + { + File.WriteAllText(_tempFile, "Test line content\nSecond line\n", Encoding.UTF8); + + using var reader = new LogfileReader( + _tempFile, + new EncodingOptions { Encoding = Encoding.UTF8 }, + multiFile: false, + bufferCount: 100, + linesPerBuffer: 500, + new MultiFileOptions(), + ReaderType.System, + PluginRegistry.PluginRegistry.Instance, + maximumLineLength: 500, + progressReporter: Core.Classes.Log.ProgressReporters.NullProgressReporter.Instance); + + reader.ReadFiles(); + + var line = reader.GetLogLineMemory(0); + Assert.That(line, Is.Not.Null); + + // Verify the FullLine memory is array-backed (block-allocated), not string-backed + var success = System.Runtime.InteropServices.MemoryMarshal.TryGetArray(line.FullLine, out var segment); + Assert.That(success, Is.True, "LogLine.FullLine should be backed by a char[] block, not a string"); + } + + [Test] + public void ReadFiles_MultipleBufferRotations_AllLinesCorrect () + { + // With linesPerBuffer=50 and 500 lines, this forces 10 buffer rotations + GenerateLogFile(_tempFile, 500); + + using var reader = new LogfileReader( + _tempFile, + new EncodingOptions { Encoding = Encoding.UTF8 }, + multiFile: false, + bufferCount: 100, + linesPerBuffer: 50, + new MultiFileOptions(), + ReaderType.System, + PluginRegistry.PluginRegistry.Instance, + maximumLineLength: 500, + progressReporter: Core.Classes.Log.ProgressReporters.NullProgressReporter.Instance); + + reader.ReadFiles(); + + Assert.That(reader.LineCount, Is.EqualTo(500)); + + // Spot-check lines across buffer boundaries + for (var i = 0; i < 500; i += 49) // stride of 49 crosses buffer boundaries + { + VerifyLine(reader, i, i); + } + } + + private static void VerifyLine (LogfileReader reader, int lineNum, int expectedIndex) + { + var line = reader.GetLogLineMemory(lineNum); + Assert.That(line, Is.Not.Null, $"Line {lineNum} should not be null"); + var text = line.FullLine.Span.ToString(); + Assert.That(text, Does.Contain($"message {expectedIndex}"), + $"Line {lineNum} content mismatch: '{text}'"); + } + + private static void GenerateLogFile (string path, int lineCount) + { + using var writer = new StreamWriter(path, false, Encoding.UTF8, bufferSize: 65536); + for (var i = 0; i < lineCount; i++) + { + writer.Write("2026-04-23 12:00:00."); + writer.Write(i % 1000); + writer.Write(" [INFO] Thread-1 SomeClass - log message "); + writer.WriteLine(i); + } + } +} \ No newline at end of file diff --git a/src/LogExpert.Tests/StreamReaderTests/PositionAwareStreamReaderDirectTests.cs b/src/LogExpert.Tests/StreamReaderTests/PositionAwareStreamReaderDirectTests.cs new file mode 100644 index 00000000..3fbbc8ee --- /dev/null +++ b/src/LogExpert.Tests/StreamReaderTests/PositionAwareStreamReaderDirectTests.cs @@ -0,0 +1,290 @@ +using System.Text; + +using LogExpert.Core.Classes.Log.Streamreaders; +using LogExpert.Core.Entities; + +using NUnit.Framework; + +namespace LogExpert.Tests.StreamReaderTests; + +[TestFixture] +public class PositionAwareStreamReaderDirectTests +{ + private static MemoryStream CreateStream (string text, Encoding? encoding = null) + { + return new MemoryStream((encoding ?? Encoding.UTF8).GetBytes(text)); + } + + #region Basic line reading + + [Test] + [TestCase("Line 1\nLine 2\nLine 3", 3)] + [TestCase("Line 1\nLine 2\nLine 3\n", 3)] + [TestCase("Line 1\r\nLine 2\r\nLine 3", 3)] + [TestCase("Line 1\r\nLine 2\r\nLine 3\r\n", 3)] + public void TryReadLine_CorrectLineCount (string text, int expectedLines) + { + using var stream = CreateStream(text); + using var reader = new PositionAwareStreamReaderDirect(stream, new EncodingOptions(), 500); + var count = 0; + while (reader.TryReadLine(out _)) + { + count++; + } + + Assert.That(count, Is.EqualTo(expectedLines)); + } + + [Test] + [TestCase("Line 1\nLine 2\nLine 3")] + [TestCase("Line 1\r\nLine 2\r\nLine 3")] + public void TryReadLine_CorrectContent (string text) + { + using var stream = CreateStream(text); + using var reader = new PositionAwareStreamReaderDirect(stream, new EncodingOptions(), 500); + + Assert.That(reader.TryReadLine(out var line1), Is.True); + Assert.That(line1.Span.ToString(), Is.EqualTo("Line 1")); + + Assert.That(reader.TryReadLine(out var line2), Is.True); + Assert.That(line2.Span.ToString(), Is.EqualTo("Line 2")); + + Assert.That(reader.TryReadLine(out var line3), Is.True); + Assert.That(line3.Span.ToString(), Is.EqualTo("Line 3")); + + Assert.That(reader.TryReadLine(out _), Is.False); + } + + [Test] + public void TryReadLine_EmptyFile_ReturnsFalse () + { + using var stream = CreateStream(""); + using var reader = new PositionAwareStreamReaderDirect(stream, new EncodingOptions(), 500); + + Assert.That(reader.TryReadLine(out _), Is.False); + } + + [Test] + public void TryReadLine_EmptyLines_Preserved () + { + using var stream = CreateStream("\n\n\n"); + using var reader = new PositionAwareStreamReaderDirect(stream, new EncodingOptions(), 500); + + var count = 0; + while (reader.TryReadLine(out var line)) + { + Assert.That(line.Length, Is.EqualTo(0)); + count++; + } + + Assert.That(count, Is.EqualTo(3)); + } + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Tests")] + public void TryReadLine_NoTrailingNewline_EmitsLastLine () + { + using var stream = CreateStream("line1\nline2"); + using var reader = new PositionAwareStreamReaderDirect(stream, new EncodingOptions(), 500); + + Assert.That(reader.TryReadLine(out var line1), Is.True); + Assert.That(line1.Span.ToString(), Is.EqualTo("line1")); + + Assert.That(reader.TryReadLine(out var line2), Is.True); + Assert.That(line2.Span.ToString(), Is.EqualTo("line2")); + + Assert.That(reader.TryReadLine(out _), Is.False); + } + + #endregion + + #region MaximumLineLength + + [Test] + public void TryReadLine_LongLine_Truncated () + { + var longLine = new string('x', 1000); + using var stream = CreateStream(longLine + "\nshort\n"); + using var reader = new PositionAwareStreamReaderDirect(stream, new EncodingOptions(), 100); + + Assert.That(reader.TryReadLine(out var line1), Is.True); + Assert.That(line1.Length, Is.EqualTo(100)); + + Assert.That(reader.TryReadLine(out var line2), Is.True); + Assert.That(line2.Span.ToString(), Is.EqualTo("short")); + } + + #endregion + + #region Cross-block boundary + + [Test] + public void TryReadLine_LineCrossesBlockBoundary_Correct () + { + // Create content that forces a line to cross the internal block boundary. + // Use a known block size and a line longer than remaining space. + var sb = new StringBuilder(); + // Fill near the block boundary with short lines, then one long line + for (var i = 0; i < 400; i++) + { + _ = sb.AppendLine(new string('A', 80)); // ~81 chars per line with \n + } + // At this point we're ~32,400 chars in. Add a long line that crosses the boundary. + var crossingLine = new string('B', 500); + _ = sb.AppendLine(crossingLine); + _ = sb.AppendLine("final line"); + + var text = sb.ToString(); + using var stream = CreateStream(text); + using var reader = new PositionAwareStreamReaderDirect(stream, new EncodingOptions(), 1000); + + // Read all lines and verify the crossing line + var lineNum = 0; + var foundCrossing = false; + while (reader.TryReadLine(out ReadOnlyMemory line)) + { + if (lineNum >= 400 && line.Length == 500) + { + Assert.That(line.Span.ToString(), Is.EqualTo(crossingLine)); + foundCrossing = true; + } + + lineNum++; + } + + Assert.That(foundCrossing, Is.True, "Should have found the 500-char crossing line"); + } + + #endregion + + #region Position parity with System reader + + [Test] + [TestCase("Line 1\nLine 2\nLine 3\n")] + [TestCase("Line 1\r\nLine 2\r\nLine 3\r\n")] + [TestCase("Short\nMedium line here\nA much longer line with more content\n")] + public void Position_MatchesSystemReader_UTF8 (string text) + { + ComparePositions(text, Encoding.UTF8); + } + + [Test] + [TestCase("Line 1\nLine 2\nLine 3\n")] + public void Position_MatchesSystemReader_ASCII (string text) + { + ComparePositions(text, Encoding.ASCII); + } + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1303:Do not pass literals as localized parameters", Justification = "Unit Tests")] + public void Position_MatchesSystemReader_UTF8_MultibyteChars () + { + var text = "Héllo wörld\nCafé résumé\nこんにちは\n"; + ComparePositions(text, Encoding.UTF8); + } + + private static void ComparePositions (string text, Encoding encoding) + { + var bytes = encoding.GetBytes(text); + + using var s1 = new MemoryStream(bytes); + using var s2 = new MemoryStream(bytes); + using var systemReader = new PositionAwareStreamReaderSystem(s1, new EncodingOptions { Encoding = encoding }, 500); + using var directReader = new PositionAwareStreamReaderDirect(s2, new EncodingOptions { Encoding = encoding }, 500); + + var lineNum = 0; + while (true) + { + var systemHasLine = systemReader.TryReadLine(out var systemLine); + var directHasLine = directReader.TryReadLine(out var directLine); + + Assert.That(directHasLine, Is.EqualTo(systemHasLine), $"Line {lineNum}: EOF mismatch"); + + if (!systemHasLine) + { + break; + } + + Assert.That(directLine.Span.ToString(), Is.EqualTo(systemLine.Span.ToString()), + $"Line {lineNum}: content mismatch"); + Assert.That(directReader.Position, Is.EqualTo(systemReader.Position), + $"Line {lineNum}: position mismatch (System={systemReader.Position}, Direct={directReader.Position})"); + + lineNum++; + } + } + + #endregion + + #region Content parity with System reader + + [Test] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Security", "CA5394:Do not use insecure randomness", Justification = "Unit Tests")] + [System.Diagnostics.CodeAnalysis.SuppressMessage("Globalization", "CA1305:Specify IFormatProvider", Justification = "Unit Tests")] + public void ContentParity_LargeFile () + { + var sb = new StringBuilder(); + var rng = new Random(42); + for (var i = 0; i < 10_000; i++) + { + _ = sb.Append("2026-04-23 12:00:00."); + _ = sb.Append(i % 1000); + _ = sb.Append(" [INFO] Thread-"); + _ = sb.Append(rng.Next(1, 32)); + _ = sb.Append(" SomeClass - Log message number "); + _ = sb.AppendLine(i.ToString()); + } + + var text = sb.ToString(); + var bytes = Encoding.UTF8.GetBytes(text); + + using var s1 = new MemoryStream(bytes); + using var s2 = new MemoryStream(bytes); + using var systemReader = new PositionAwareStreamReaderSystem(s1, new EncodingOptions { Encoding = Encoding.UTF8 }, 500); + using var directReader = new PositionAwareStreamReaderDirect(s2, new EncodingOptions { Encoding = Encoding.UTF8 }, 500); + + var lineNum = 0; + while (systemReader.TryReadLine(out var systemLine)) + { + Assert.That(directReader.TryReadLine(out var directLine), Is.True, $"Direct reader ended early at line {lineNum}"); + Assert.That(directLine.Span.ToString(), Is.EqualTo(systemLine.Span.ToString()), $"Line {lineNum}: content mismatch"); + lineNum++; + } + + Assert.That(directReader.TryReadLine(out _), Is.False, "Direct reader has extra lines"); + Assert.That(lineNum, Is.EqualTo(10_000)); + } + + #endregion + + #region DetachBlocks + + [Test] + public void DetachBlocks_ReturnsCompletedBlocks () + { + // Create enough content to fill multiple blocks + var sb = new StringBuilder(); + for (var i = 0; i < 1000; i++) + { + _ = sb.AppendLine(new string('X', 80)); + } + + using var stream = CreateStream(sb.ToString()); + using var reader = new PositionAwareStreamReaderDirect(stream, new EncodingOptions(), 500); + + // Read all lines + while (reader.TryReadLine(out _)) + { + // Intentionally empty: consume all lines to advance reader state. + } + + var blocks = reader.DetachBlocks(); + Assert.That(blocks.Count, Is.GreaterThan(0), "Should have completed blocks to detach"); + + // Second detach should be empty + var blocks2 = reader.DetachBlocks(); + Assert.That(blocks2.Count, Is.EqualTo(0)); + } + + #endregion +} \ No newline at end of file diff --git a/src/LogExpert.UI/Controls/LogWindow/ColumnCache.cs b/src/LogExpert.UI/Controls/LogWindow/ColumnCache.cs index 54887a63..b663d861 100644 --- a/src/LogExpert.UI/Controls/LogWindow/ColumnCache.cs +++ b/src/LogExpert.UI/Controls/LogWindow/ColumnCache.cs @@ -1,7 +1,8 @@ using ColumnizerLib; using LogExpert.Core.Callback; -using LogExpert.Core.Classes.Log; +using LogExpert.Core.Classes.Log.Buffers; +using LogExpert.Core.Interfaces; namespace LogExpert.UI.Controls.LogWindow; @@ -17,6 +18,7 @@ internal class ColumnCache private ILogLineMemory[] _prefetchedLines; private int _prefetchStartLine = -1; private int _prefetchCount; + private PinHandle? _pinHandle; #endregion @@ -25,17 +27,48 @@ internal class ColumnCache /// /// Prefetch a range of lines in a single batch call. /// Call this before a paint cycle with the visible row range. + /// Pin-before-read: pins are acquired BEFORE reading lines to prevent + /// the GC eviction thread from returning char blocks between read and pin. /// - internal void Prefetch (LogfileReader logFileReader, int startLine, int count) + internal void Prefetch (ILogfileReader logFileReader, int startLine, int count) { if (startLine == _prefetchStartLine && count == _prefetchCount) { return; // already prefetched this exact range } - _prefetchedLines = logFileReader.GetLogLineMemories(startLine, count); + //Pin BEFORE reading — this should prevent GC thread from evicting buffers + //between GetLogLineMemories() and PinRange(). + PinHandle? newHandle = null; + if (logFileReader is IBufferPinning pinning) + { + newHandle = pinning.PinRange(startLine, startLine + count - 1); + } + + var newLines = logFileReader.GetLogLineMemories(startLine, count); + + //Atomic swap: old handle released AFTER new is in place + var oldHandle = _pinHandle; + _pinHandle = newHandle; + _prefetchedLines = newLines; _prefetchStartLine = startLine; - _prefetchCount = _prefetchedLines.Length; + _prefetchCount = newLines.Length; + + // Now safe to release old pins — new pins cover the new visible range + oldHandle?.Dispose(); + } + + /// + /// Returns the prefetched line for the given line number if it falls within the + /// currently pinned range. Returns null if not available. + /// + internal ILogLineMemory? GetPrefetchedLine (int lineNumber) + { + return _prefetchedLines != null + && lineNumber >= _prefetchStartLine + && lineNumber < _prefetchStartLine + _prefetchCount + ? _prefetchedLines[lineNumber - _prefetchStartLine] + : null; } /// @@ -43,13 +76,29 @@ internal void Prefetch (LogfileReader logFileReader, int startLine, int count) /// internal void InvalidatePrefetch () { + _pinHandle?.Dispose(); + _pinHandle = null; + _prefetchedLines = null; _prefetchStartLine = -1; _prefetchCount = 0; _lastLineNumber = -1; } - internal IColumnizedLogLineMemory GetColumnsForLine (LogfileReader logFileReader, int lineNumber, ILogLineMemoryColumnizer columnizer, ColumnizerCallback columnizerCallback) + /// + /// Forces the next call to re-fetch and re-pin, without releasing + /// current pins. Use this instead of when the visible data + /// may be stale but the underlying buffers must remain pinned until replacement pins are acquired. + /// + internal void MarkPrefetchStale () + { + _prefetchStartLine = -1; + _prefetchCount = 0; + _lastLineNumber = -1; + _cachedColumns = null; + } + + internal IColumnizedLogLineMemory GetColumnsForLine (ILogfileReader logFileReader, int lineNumber, ILogLineMemoryColumnizer columnizer, ColumnizerCallback columnizerCallback) { if (_lastColumnizer != columnizer || (_lastLineNumber != lineNumber && _cachedColumns != null) || columnizerCallback.LineNum != lineNumber) { @@ -65,6 +114,9 @@ internal IColumnizedLogLineMemory GetColumnsForLine (LogfileReader logFileReader line = _prefetchedLines[lineNumber - _prefetchStartLine]; } + // Fallback: read directly. This is safe because the caller (CellValueNeeded) + // has already called Prefetch/PrefetchFilterVisibleLines which pins the relevant + // buffers. Pinned buffers won't have their char blocks returned to the pool. line ??= logFileReader.GetLogLineMemoryWithWait(lineNumber).Result; if (line != null) diff --git a/src/LogExpert.UI/Controls/LogWindow/LogWindow.cs b/src/LogExpert.UI/Controls/LogWindow/LogWindow.cs index 238eba6e..18cb1030 100644 --- a/src/LogExpert.UI/Controls/LogWindow/LogWindow.cs +++ b/src/LogExpert.UI/Controls/LogWindow/LogWindow.cs @@ -1,5 +1,6 @@ using System.ComponentModel; using System.Globalization; +using System.Runtime.InteropServices; using System.Runtime.Versioning; using System.Text; using System.Text.RegularExpressions; @@ -103,6 +104,7 @@ internal partial class LogWindow : DockContent, ILogPaintContextUI, ILogView, IL private readonly Lock _timeSyncListLock = new(); private ColumnCache _columnCache = new(); + private ColumnCache _filterColumnCache = new(); private readonly StringFormat _format = new() { @@ -418,7 +420,7 @@ public bool IsMultiFile public Font BoldFont { get; private set; } - LogfileReader ILogWindow.LogFileReader => _logFileReader; + ILogfileReader ILogWindow.LogFileReader => _logFileReader; //public event EventHandler ILogWindow.FileSizeChanged //{ @@ -455,24 +457,6 @@ public Bookmark GetBookmarkForLine (int lineNum) #region Internals - internal IColumnizedLogLineMemory GetColumnsForLine (int lineNumber) - { - return _columnCache.GetColumnsForLine(_logFileReader, lineNumber, CurrentColumnizer, ColumnizerCallbackObject); - - //string line = this.logFileReader.GetLogLine(lineNumber); - //if (line != null) - //{ - // string[] cols; - // this.columnizerCallback.LineNum = lineNumber; - // cols = this.CurrentColumnizer.SplitLine(this.columnizerCallback, line); - // return cols; - //} - //else - //{ - // return null; - //} - } - #region Apply Resources private void SetResources () @@ -998,6 +982,49 @@ private void PrefetchVisibleLines () } } + private void PrefetchFilterVisibleLines () + { + if (_logFileReader == null || _filterResultList == null || _filterResultList.Count == 0) + { + return; + } + + var firstVisible = filterGridView.FirstDisplayedScrollingRowIndex; + var visibleCount = filterGridView.DisplayedRowCount(includePartialRow: true); + + if (firstVisible < 0 || visibleCount <= 0) + { + return; + } + + // The filter grid maps grid rows -> original line numbers. + // We need to pin the actual line buffers for the visible filter rows. + var endVisible = Math.Min(firstVisible + visibleCount, _filterResultList.Count); + var span = CollectionsMarshal.AsSpan(_filterResultList)[firstVisible..endVisible]; + + var minLine = span[0]; + var maxLine = minLine; + + for (int i = 0; i < span.Length; i++) + { + int lineNum = span[i]; + + if (lineNum < minLine) + { + minLine = lineNum; + } + else if (lineNum > maxLine) + { + maxLine = lineNum; + } + } + + // Prefetch the tight range covering only the visible filter rows. + // For sorted filter results this is the same as before but + // bounded to actual visible rows, not the entire result set. + _filterColumnCache.Prefetch(_logFileReader, minLine, maxLine - minLine + 1); + } + [SupportedOSPlatform("windows")] private void OnDataGridViewCellValuePushed (object sender, DataGridViewCellValueEventArgs e) { @@ -1019,10 +1046,10 @@ private void OnDataGridViewCellValuePushed (object sender, DataGridViewCellValue var oldValue = cols.ColumnValues[e.ColumnIndex - 2].FullValue; var newValue = (string)e.Value; - //string oldValue = (string) this.dataGridView.Rows[e.RowIndex].Cells[e.ColumnIndex].Value; - //TODO OLD VALUE needs to be ReadOnlySpan - CurrentColumnizer.PushValue(ColumnizerCallbackObject, e.ColumnIndex - 2, newValue, oldValue.ToString()); + + CurrentColumnizer.PushValue(ColumnizerCallbackObject, e.ColumnIndex - 2, newValue, oldValue); dataGridView.Refresh(); + TimeSpan timeSpan = new(CurrentColumnizer.GetTimeOffset() * TimeSpan.TicksPerMillisecond); var span = timeSpan.ToString(); var index = span.LastIndexOf('.'); @@ -1057,9 +1084,6 @@ private void OnDataGridViewCurrentCellChanged (object sender, EventArgs e) { SyncTimestampDisplay(); } - - //MethodInvoker invoker = new MethodInvoker(DisplayCurrentFileOnStatusline); - //invoker.BeginInvoke(null, null); } } @@ -1127,8 +1151,10 @@ private void OnFilterGridViewCellValueNeeded (object sender, DataGridViewCellVal return; } + PrefetchFilterVisibleLines(); + var lineNum = _filterResultList[e.RowIndex]; - e.Value = GetCellValue(lineNum, e.ColumnIndex); + e.Value = GetFilterCellValue(lineNum, e.ColumnIndex); } [SupportedOSPlatform("windows")] @@ -3079,6 +3105,10 @@ private void UpdateGrid (LogEventArgs logEventArgs) if (_guiStateArgs.FollowTail && dataGridView.RowCount > 0) { + // Mark prefetch stale so the next paint re-fetches fresh data. + // Do NOT call InvalidatePrefetch() here — it unpins buffers, creating a + // window where the GC thread can evict and return blocks to the pool. + _columnCache.MarkPrefetchStale(); dataGridView.FirstDisplayedScrollingRowIndex = dataGridView.RowCount - 1; OnTailFollowed(EventArgs.Empty); } @@ -5220,7 +5250,7 @@ private void CopyMarkedLinesToClipboard () _ = clipText.AppendLine(line.ToClipBoardText()); } - Clipboard.SetText(clipText.ToString()); + Clipboard.SetDataObject(clipText.ToString()); } } @@ -6171,6 +6201,7 @@ public void LoadFile (string fileName, EncodingOptions encodingOptions) } _columnCache = new ColumnCache(); + _filterColumnCache = new ColumnCache(); try { @@ -6236,6 +6267,7 @@ public void LoadFilesAsMulti (string[] fileNames, EncodingOptions encodingOption EncodingOptions = encodingOptions; _columnCache = new ColumnCache(); + _filterColumnCache = new ColumnCache(); _logFileReader = new(fileNames, EncodingOptions, Preferences.BufferCount, Preferences.LinesPerBuffer, _multiFileOptions, Preferences.ReaderType, PluginRegistry.PluginRegistry.Instance, ConfigManager.Settings.Preferences.MaxLineLength); @@ -6477,7 +6509,7 @@ public IColumnMemory GetCellValue (int rowIndex, int columnIndex) try { - var cols = GetColumnsForLine(rowIndex); + var cols = _columnCache.GetColumnsForLine(_logFileReader, rowIndex, CurrentColumnizer, ColumnizerCallbackObject); if (cols != null && cols.ColumnValues != null) { if (columnIndex <= cols.ColumnValues.Length + 1) @@ -6494,10 +6526,125 @@ public IColumnMemory GetCellValue (int rowIndex, int columnIndex) : Column.EmptyColumn; } } - catch +#if DEBUG + catch (IndexOutOfRangeException ex) { + + _logger.Warn(ex, "Failed to get cell value due to index error. rowIndex={RowIndex}, columnIndex={ColumnIndex}", rowIndex, columnIndex); return Column.EmptyColumn; } +#else + catch (IndexOutOfRangeException) + { + return Column.EmptyColumn; + + } +#endif +#if DEBUG + catch (ArgumentOutOfRangeException ex) + { + + _logger.Warn(ex, "Failed to get cell value due to argument range error. rowIndex={RowIndex}, columnIndex={ColumnIndex}", rowIndex, columnIndex); + return Column.EmptyColumn; + } +#else + catch (ArgumentOutOfRangeException) + { + return Column.EmptyColumn; + } +#endif +#if DEBUG + catch (NullReferenceException ex) + { + + _logger.Warn(ex, "Failed to get cell value due to null state. rowIndex={RowIndex}, columnIndex={ColumnIndex}", rowIndex, columnIndex); + return Column.EmptyColumn; + } +#else + catch (NullReferenceException) + { + return Column.EmptyColumn; + } +#endif + return Column.EmptyColumn; + } + + /// + /// Filter grid variant of GetCellValue that uses _filterColumnCache (which pins + /// the filter's visible buffers) instead of _columnCache (which pins the main grid's range). + /// + private IColumnMemory GetFilterCellValue (int rowIndex, int columnIndex) + { + if (columnIndex == 1) + { + return new Column + { + FullValue = $"{rowIndex + 1}".AsMemory() // line number + }; + } + + if (columnIndex == 0) + { + return Column.EmptyColumn; + } + + try + { + var cols = _filterColumnCache.GetColumnsForLine(_logFileReader, rowIndex, CurrentColumnizer, ColumnizerCallbackObject); + if (cols != null && cols.ColumnValues != null) + { + if (columnIndex <= cols.ColumnValues.Length + 1) + { + var value = cols.ColumnValues[columnIndex - 2]; + + return value != null && !value.DisplayValue.IsEmpty + ? value + : value; + } + + return columnIndex == 2 + ? cols.ColumnValues[^1] + : Column.EmptyColumn; + } + } +#if DEBUG + catch (IndexOutOfRangeException ex) + { + + _logger.Warn(ex, "Failed to get filter cell value due to index error. rowIndex={RowIndex}, columnIndex={ColumnIndex}", rowIndex, columnIndex); + return Column.EmptyColumn; + } +#else + catch (IndexOutOfRangeException) + { + return Column.EmptyColumn; + } +#endif +#if DEBUG + catch (ArgumentOutOfRangeException ex) + { + + _logger.Warn(ex, "Failed to get filter cell value due to argument range error. rowIndex={RowIndex}, columnIndex={ColumnIndex}", rowIndex, columnIndex); + return Column.EmptyColumn; + } +#else + catch (ArgumentOutOfRangeException) + { + return Column.EmptyColumn; + } +#endif +#if DEBUG + catch (NullReferenceException ex) + { + _logger.Warn(ex, "Failed to get filter cell value due to null state. rowIndex={RowIndex}, columnIndex={ColumnIndex}", rowIndex, columnIndex); + return Column.EmptyColumn; + } +#else + catch (NullReferenceException) + { + return Column.EmptyColumn; + } +#endif return Column.EmptyColumn; } @@ -6515,7 +6662,23 @@ public void CellPainting (bool focused, int rowIndex, int columnIndex, bool isFi rowIndex = _filterResultList[rowIndex]; } - var line = _logFileReader.GetLogLineMemoryWithWait(rowIndex).Result; + // Ensure prefetch is current — CellPainting fires BEFORE CellValueNeeded on scroll jumps, + // so the prefetch may still be at the old range. Without this, GetPrefetchedLine returns null + // and the fallback (GetLogLineMemoryWithWait) fetches an unprotected line whose backing block + // can be returned to the shared ArrayPool and rented by another window's reader. + if (!isFilteredGridView) + { + PrefetchVisibleLines(); + } + else + { + PrefetchFilterVisibleLines(); + } + + // Use only prefetched (pinned) data — no unprotected fallback + ILogLineMemory line = !isFilteredGridView + ? _columnCache.GetPrefetchedLine(rowIndex) + : _filterColumnCache.GetPrefetchedLine(rowIndex); if (line == null) { @@ -6666,12 +6829,14 @@ public void FollowTailChanged (bool isChecked, bool byTrigger) { if (dataGridView.RowCount >= _logFileReader.LineCount && _logFileReader.LineCount > 0) { + // Mark stale instead of invalidating — keeps old buffers pinned until + // the next Prefetch atomically swaps in new pins. + _columnCache.MarkPrefetchStale(); dataGridView.FirstDisplayedScrollingRowIndex = _logFileReader.LineCount - 1; } } _ = BeginInvoke(new MethodInvoker(dataGridView.Refresh)); - //this.dataGridView.Refresh(); _logWindowCoordinator.NotifyFollowTailChanged(this, isChecked, byTrigger); SendGuiStateUpdate(); } @@ -7772,10 +7937,10 @@ public int FindTimestampLineInternal (int lineNum, int rangeStart, int rangeEnd, } /** - * Get the timestamp for the given line number. If the line - * has no timestamp, the previous line will be checked until a - * timestamp is found. - */ + * Get the timestamp for the given line number. If the line + * has no timestamp, the previous line will be checked until a + * timestamp is found. +*/ public (DateTime timeStamp, int lastLineNumber) GetTimestampForLine (int lastLineNum, bool roundToSeconds) { lock (_currentColumnizerLock) @@ -7834,10 +7999,10 @@ public int FindTimestampLineInternal (int lineNum, int rangeStart, int rangeEnd, } /** - * Get the timestamp for the given line number. If the line - * has no timestamp, the next line will be checked until a - * timestamp is found. - */ + * Get the timestamp for the given line number. If the line + * has no timestamp, the next line will be checked until a + * timestamp is found. +*/ public DateTime GetTimestampForLineForward (ref int lineNum, bool roundToSeconds) { lock (_currentColumnizerLock) diff --git a/src/LogExpert.UI/Dialogs/SettingsDialog.cs b/src/LogExpert.UI/Dialogs/SettingsDialog.cs index 7aed69d7..ad8671dc 100644 --- a/src/LogExpert.UI/Dialogs/SettingsDialog.cs +++ b/src/LogExpert.UI/Dialogs/SettingsDialog.cs @@ -250,7 +250,10 @@ private void FillReaderTypeList () { foreach (var readerType in Enum.GetValues()) { - _ = comboBoxReaderType.Items.Add(readerType); + if (!comboBoxReaderType.Items.Contains(readerType)) + { + _ = comboBoxReaderType.Items.Add(readerType); + } } comboBoxReaderType.SelectedItem = Preferences.ReaderType; diff --git a/src/LogExpert.UI/Services/FileOperationService/FileOperationService.cs b/src/LogExpert.UI/Services/FileOperationService/FileOperationService.cs index 563338dc..1e4bae80 100644 --- a/src/LogExpert.UI/Services/FileOperationService/FileOperationService.cs +++ b/src/LogExpert.UI/Services/FileOperationService/FileOperationService.cs @@ -331,8 +331,8 @@ public void LoadStartupFiles (IList lastOpenFiles, string[]? startupFile { _ = AddFileTab(new FileTabRequest { FileName = name }); } - } - _configManager.ClearLastOpenFilesList(); + _configManager.ClearLastOpenFilesList(); + } } } \ No newline at end of file diff --git a/src/LogExpert.sln b/src/LogExpert.sln index 43437357..ee967ce8 100644 --- a/src/LogExpert.sln +++ b/src/LogExpert.sln @@ -24,6 +24,7 @@ EndProject Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{DE6375A4-B4C4-4620-8FFB-B9D5A4E21144}" ProjectSection(SolutionItems) = preProject .editorconfig = .editorconfig + ..\.gitignore = ..\.gitignore Solution Items\AssemblyInfo.cs = Solution Items\AssemblyInfo.cs ..\CHANGELOG.md = ..\CHANGELOG.md ..\.github\copilot-instructions.md = ..\.github\copilot-instructions.md @@ -84,6 +85,7 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "docs", "docs", "{39822C1B-E ProjectSection(SolutionItems) = preProject docs\PLUGIN_DEVELOPMENT_GUIDE.md = docs\PLUGIN_DEVELOPMENT_GUIDE.md docs\PLUGIN_HASH_MANAGEMENT.md = docs\PLUGIN_HASH_MANAGEMENT.md + docs\UBIQUITOUS_LANGUAGE.md = docs\UBIQUITOUS_LANGUAGE.md EndProjectSection EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "PluginHashGenerator.Tool", "PluginHashGenerator.Tool\PluginHashGenerator.Tool.csproj", "{B2E6D6E0-C995-8F9D-4482-869C6A5A64C0}" @@ -100,11 +102,6 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "LogExpert.Persister.Tests", EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "LogExpert.Benchmarks", "LogExpert.Benchmarks\LogExpert.Benchmarks.csproj", "{1046779B-500D-8260-33BA-BC778C4B836F}" EndProject -Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "performance", "performance", "{C83F15B6-F6E0-4526-A5C5-47806772E49A}" - ProjectSection(SolutionItems) = preProject - docs\performance\BENCHMARK_SUMMARY.md = docs\performance\BENCHMARK_SUMMARY.md - EndProjectSection -EndProject Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "help", "help", "{9FC7BC64-CE77-45DF-B8AC-10F1D0336E76}" ProjectSection(SolutionItems) = preProject HelpSmith\LogExpert.chm = HelpSmith\LogExpert.chm @@ -242,7 +239,6 @@ Global {39822C1B-E4C6-40F3-86C4-74C68BDEF3D0} = {DE6375A4-B4C4-4620-8FFB-B9D5A4E21144} {CAD17410-CE8C-4FE5-91DE-1B3DE2945135} = {848C24BA-BEBA-48EC-90E6-526ECAB6BB4A} {1046779B-500D-8260-33BA-BC778C4B836F} = {848C24BA-BEBA-48EC-90E6-526ECAB6BB4A} - {C83F15B6-F6E0-4526-A5C5-47806772E49A} = {39822C1B-E4C6-40F3-86C4-74C68BDEF3D0} {9FC7BC64-CE77-45DF-B8AC-10F1D0336E76} = {DE6375A4-B4C4-4620-8FFB-B9D5A4E21144} EndGlobalSection GlobalSection(ExtensibilityGlobals) = postSolution diff --git a/src/PluginRegistry/PluginHashGenerator.Generated.cs b/src/PluginRegistry/PluginHashGenerator.Generated.cs index 9164253c..3e31f327 100644 --- a/src/PluginRegistry/PluginHashGenerator.Generated.cs +++ b/src/PluginRegistry/PluginHashGenerator.Generated.cs @@ -10,7 +10,7 @@ public static partial class PluginValidator { /// /// Gets pre-calculated SHA256 hashes for built-in plugins. - /// Generated: 2026-04-21 17:29:42 UTC + /// Generated: 2026-05-04 10:41:14 UTC /// Configuration: Release /// Plugin count: 22 /// @@ -18,28 +18,28 @@ public static Dictionary GetBuiltInPluginHashes() { return new Dictionary(StringComparer.OrdinalIgnoreCase) { - ["AutoColumnizer.dll"] = "8E966C2C6671A03093867E2104C7C4E24F9D866F448AA5401F053F8BED23AFE1", + ["AutoColumnizer.dll"] = "FC795D0F77912FC8CA521912072A33A4526B5DD417091EE18F06191E02E019FD", ["BouncyCastle.Cryptography.dll"] = "E5EEAF6D263C493619982FD3638E6135077311D08C961E1FE128F9107D29EBC6", ["BouncyCastle.Cryptography.dll (x86)"] = "E5EEAF6D263C493619982FD3638E6135077311D08C961E1FE128F9107D29EBC6", - ["CsvColumnizer.dll"] = "DCFCDBB094D328F246FD93809F4512C336B28584C4D1E7F13E73DDA26587572D", - ["CsvColumnizer.dll (x86)"] = "DCFCDBB094D328F246FD93809F4512C336B28584C4D1E7F13E73DDA26587572D", - ["DefaultPlugins.dll"] = "853FFDDD12B3F554E40476CE4E7C8A1543295FFEAFC445F31BD89DCC42597505", - ["FlashIconHighlighter.dll"] = "127F1C2C9E3FA58D24D52487BFBB69A44CDA5DE6A4AEBD2D61A259165C95E2F5", - ["GlassfishColumnizer.dll"] = "ECCE8D6B24DDD1016DF14F5DE8155FD2B92F27EBC877CDE9A7B0148C90AE60C5", - ["JsonColumnizer.dll"] = "C64E0E16204EEC4C012F4D02257A77A165EE1F3CE1A06EA3F525F6D005EF2B50", - ["JsonCompactColumnizer.dll"] = "849D1B630232C66F8C6311B82F0719D1F97714E55951CF38E8D696EFB39ACCD7", - ["Log4jXmlColumnizer.dll"] = "F1D80A33E4B61CCB20184BDA7CB141C4A481C9E780FE11408E09F475F5A2B9C3", - ["LogExpert.Core.dll"] = "1E1655F42A545573A908031FD5BB34FD434ECF377602D79A1CEFA87757746F1B", - ["LogExpert.Resources.dll"] = "E7D15126016C257E5B6E04E65538CDC12809835B6C88FC4E01E1532F7561DD98", + ["CsvColumnizer.dll"] = "0BA0126080FAEC25281C2E009CECD28F9CE95B71FB5339A27ED8C2B3761BBABE", + ["CsvColumnizer.dll (x86)"] = "0BA0126080FAEC25281C2E009CECD28F9CE95B71FB5339A27ED8C2B3761BBABE", + ["DefaultPlugins.dll"] = "D836EFDF3CE6E700E1E304DF1C42FC27047F4FA97104F63CC6C83AF408ED435D", + ["FlashIconHighlighter.dll"] = "9E920F4B7A50EBA43BA1327987DB4C70D901FA2A91318B8994F81C1D0518A857", + ["GlassfishColumnizer.dll"] = "4075E5B23FB304B016A6B8C0CA976D891FA5338C2345E075DA72A21F2793FBDE", + ["JsonColumnizer.dll"] = "6551E65D7611D87B8E2C475D4FBC35D28FA5A12F0F1880C55FA31B33A14AC8B5", + ["JsonCompactColumnizer.dll"] = "99F820A8AA6A3B439AD579232FAB63B8F2934973A9761A1AB4BEEF21EE06471D", + ["Log4jXmlColumnizer.dll"] = "1C6238C0C0593675A37C9777869C9D7C3036352AEEA33C0DB8C659EEA90B4350", + ["LogExpert.Core.dll"] = "C99E67DF224EF6F3EFB81423DA0EF03CDBC035256679E628ACE57502C05CEE3D", + ["LogExpert.Resources.dll"] = "F7F10951B324469CEB20A71DFC7446EBA4F4EFD1E4F4041645D0D64DD3E2F1D7", ["Microsoft.Extensions.DependencyInjection.Abstractions.dll"] = "67FA4325000DB017DC0C35829B416F024F042D24EFB868BCF17A895EE6500A93", ["Microsoft.Extensions.DependencyInjection.Abstractions.dll (x86)"] = "67FA4325000DB017DC0C35829B416F024F042D24EFB868BCF17A895EE6500A93", ["Microsoft.Extensions.Logging.Abstractions.dll"] = "BB853130F5AFAF335BE7858D661F8212EC653835100F5A4E3AA2C66A4D4F685D", ["Microsoft.Extensions.Logging.Abstractions.dll (x86)"] = "BB853130F5AFAF335BE7858D661F8212EC653835100F5A4E3AA2C66A4D4F685D", - ["RegexColumnizer.dll"] = "E26F22FDA3882B616B30E56446AF39E7563DA846ED7646F350CCB28DF7B01D56", - ["SftpFileSystem.dll"] = "D3069CD5F9F49ABB6912D22408E6812F59B8CF302A8C89D51E9651404CEAC482", - ["SftpFileSystem.dll (x86)"] = "E503DB9BA8770A853D61686605FE8DE265A813A5A22AB33FA4BB675803D10EF2", - ["SftpFileSystem.Resources.dll"] = "D1B38ACA3ADB0326C0D24BB36CEFCE0ADEED22D09B435A4884047DDA724E8901", - ["SftpFileSystem.Resources.dll (x86)"] = "D1B38ACA3ADB0326C0D24BB36CEFCE0ADEED22D09B435A4884047DDA724E8901", + ["RegexColumnizer.dll"] = "9456D9FEC4B63EAA67CB9E89FFAA1A69062F2BB384AE67749ED77569D5BA7D27", + ["SftpFileSystem.dll"] = "E3C7EBC2A51E2159A5CC049C0827F09BC3D5A15E344B31BB748BF40EF155248B", + ["SftpFileSystem.dll (x86)"] = "6A765CB8606E866DCEE9C1DB14A00FA04B799E60FBEAA4478E01FC2133256D8C", + ["SftpFileSystem.Resources.dll"] = "9A9512846AC9E511D5562A437462525F8736E3E4A62B24458A70288EF189593F", + ["SftpFileSystem.Resources.dll (x86)"] = "9A9512846AC9E511D5562A437462525F8736E3E4A62B24458A70288EF189593F", }; } diff --git a/src/RegexColumnizer/RegexColumnizer.cs b/src/RegexColumnizer/RegexColumnizer.cs index f00f2915..b13cae87 100644 --- a/src/RegexColumnizer/RegexColumnizer.cs +++ b/src/RegexColumnizer/RegexColumnizer.cs @@ -380,6 +380,11 @@ public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, st throw new NotImplementedException(); } + public void PushValue (ILogLineMemoryColumnizerCallback callback, int column, string value, ReadOnlyMemory oldValue) + { + throw new NotImplementedException(); + } + /// /// Displays a configuration dialog for the columnizer and saves the updated settings to the specified configuration /// directory. diff --git a/src/Solution Items/AssemblyInfo.cs b/src/Solution Items/AssemblyInfo.cs index 2bbf4fba..9b16ee0a 100644 --- a/src/Solution Items/AssemblyInfo.cs +++ b/src/Solution Items/AssemblyInfo.cs @@ -1,4 +1,4 @@ using System.Runtime.InteropServices; [assembly: ComVisible(false)] -[assembly: System.Resources.NeutralResourcesLanguage("en")] \ No newline at end of file +[assembly: System.Resources.NeutralResourcesLanguage("en")] diff --git a/src/docs/UBIQUITOUS_LANGUAGE.md b/src/docs/UBIQUITOUS_LANGUAGE.md new file mode 100644 index 00000000..684ee504 --- /dev/null +++ b/src/docs/UBIQUITOUS_LANGUAGE.md @@ -0,0 +1,75 @@ +# Ubiquitous Language + +## Log File & Lines + +| Term | Definition | Aliases to avoid | +|------|-----------|-----------------| +| **Log Line** | A single textual entry read from a log file, identified by a zero-based line number | Row, record, entry | +| **Log Line Memory** (`ILogLineMemory`) | A memory-efficient representation of a Log Line's content, providing access to the full line text | LogLine, line string | +| **Log Buffer** | A fixed-size block of cached Log Lines held in memory to avoid repeated disk reads | Cache, page | +| **Logfile Reader** (`LogfileReader`) | The component that reads a log file from disk, manages Log Buffers, and tracks file position and encoding | File reader, stream reader | +| **Position-Aware Stream Reader** | A stream reader that tracks its byte position within the file to enable random access to Log Lines | StreamReader | +| **Timestamp** | The date/time parsed from a Log Line by a Columnizer, used for time-based navigation and sync | Date, time | + +## Columnizer (Plugin System) + +| Term | Definition | Aliases to avoid | +|------|-----------|-----------------| +| **Columnizer** | A plugin that splits a single Log Line into named columns for display in the data grid | Parser, splitter, formatter | +| **Column** | A single named field extracted from a Log Line by a Columnizer | Field, cell | +| **Auto Columnizer** | A Columnizer that automatically detects the log format and delegates to the appropriate Columnizer | Default parser | +| **File System Plugin** (`IFileSystemPlugin`) | A plugin that provides file data to LogExpert, selected by URI scheme (e.g., local file, SFTP) | File provider, data source | + +## Viewing & Navigation + +| Term | Definition | Aliases to avoid | +|------|-----------|-----------------| +| **Log Window** (`ILogWindow`) | A UI component that displays and manages a single log file (or multi-file set) in the data grid | View, panel, editor | +| **Log Tab Window** | The top-level MDI container that hosts multiple Log Windows as tabs | Main window, MDI parent | +| **Log Tab Page** | A single tab within the Log Tab Window, wrapping one Log Window | Tab | +| **Tail Mode** | A real-time monitoring mode where the view automatically scrolls to show newly appended Log Lines | Follow, live view, watch | +| **Multi-File Mode** | A mode where multiple log files are displayed together as one virtual file within a single Log Window | Merged view | + +## Filtering + +| Term | Definition | Aliases to avoid | +|------|-----------|-----------------| +| **Filter Params** | The set of criteria (text, regex, case sensitivity) that define which Log Lines match a filter | Filter settings, search criteria | +| **Filter Pipe** (`FilterPipe`) | A mechanism that writes matched Log Lines to a temp file and maintains a line-number mapping back to the original file | Filter stream, filter output | +| **Filter-to-Tab** | The action of piping filtered results into a new Log Tab Page for isolated viewing | Filter view | + +## Highlighting & Bookmarks + +| Term | Definition | Aliases to avoid | +|------|-----------|-----------------| +| **Highlight Entry** | A rule that matches Log Lines by text or regex and applies visual formatting (colors, bold) | Hilight entry, highlight rule | +| **Highlight Group** | A named collection of Highlight Entries that can be activated or deactivated together | Hilight group, color scheme | +| **Bookmark** | A user- or auto-generated marker on a specific Log Line, carrying optional comment text and an overlay | Marker, pin, flag | +| **Bookmark Overlay** | A draggable visual annotation displayed on the data grid at a Bookmark's position | Tooltip, popup | +| **Auto-Generated Bookmark** | A transient Bookmark created by scanning Highlight Entries with the "set bookmark" flag; not persisted | Scan bookmark | + +## Relationships + +- A **Logfile Reader** manages a collection of **Log Buffers**, each holding a block of **Log Lines** +- A **Columnizer** splits a **Log Line** into one or more **Columns** and optionally parses its **Timestamp** +- A **Log Window** uses exactly one **Logfile Reader** and one active **Columnizer** +- A **Log Tab Window** contains one or more **Log Tab Pages**, each wrapping one **Log Window** +- A **Filter Pipe** applies **Filter Params** to a **Log Window** and maps filtered lines back to original line numbers +- A **Highlight Group** contains one or more **Highlight Entries** +- A **Highlight Entry** with "set bookmark" enabled can produce **Auto-Generated Bookmarks** +- A **File System Plugin** provides the file stream consumed by a **Logfile Reader** + +## Example dialogue + +> **Dev:** "When a user switches the **Columnizer** on a **Log Window**, do the **Bookmarks** move?" +> **Domain expert:** "No — **Bookmarks** are tied to line numbers, not to **Column** content. The **Columnizer** only changes how each **Log Line** is split into **Columns** for display." +> **Dev:** "And the **Filter Pipe** — does it re-run when the **Columnizer** changes?" +> **Domain expert:** "Not automatically. The **Filter Pipe** operates on the raw **Log Line** text using **Filter Params**. Switching the **Columnizer** doesn't invalidate the filter results." +> **Dev:** "What about **Tail Mode**? Does new content flow through an active **Filter Pipe**?" +> **Domain expert:** "Yes. In **Tail Mode**, the **Logfile Reader** detects appended **Log Lines**, and any active **Filter Pipe** evaluates them against its **Filter Params** in real time." + +## Flagged ambiguities + +- "Hilight" vs "Highlight" — the codebase historically used the misspelling "hilight" (e.g., `HilightEntryList`). The canonical term is **Highlight Entry** and **Highlight Group**. Legacy property names with the typo exist only for backward compatibility with old settings files. +- "Account" is not used in this domain; "User" refers to the person operating LogExpert, not an authentication identity. +- "Entry" is overloaded — it can mean a **Log Line** (log entry), a **Highlight Entry** (highlight rule), or an **Action Entry** (trigger action). Always qualify: **Log Line**, **Highlight Entry**, or **Action Entry**. diff --git a/src/favorites.json b/src/favorites.json new file mode 100644 index 00000000..80d81225 --- /dev/null +++ b/src/favorites.json @@ -0,0 +1,4 @@ +{ + "version": 2, + "items": [] +} \ No newline at end of file