Performance Benchmark CLI - Report Formats¶

Overview¶

The Performance Benchmark CLI supports multiple report formats, each designed for specific use cases. This document details all available report formats, their features, configuration options, and best practices.

Available Report Formats¶

JSON - Machine-readable, ideal for CI/CD and programmatic analysis
HTML - Interactive web reports with charts and detailed metrics
PDF - Printable reports for documentation and presentations
Markdown - Documentation-friendly format for GitHub/wiki
Console - Rich terminal output for interactive sessions

Generating Reports¶

During Benchmark Execution¶

Automatically generate reports when running benchmarks:

# Default formats (from config)
dotnet run -- run

# Specify formats
dotnet run -- run --format html,pdf,markdown

# Console only (no files)
dotnet run -- run --format console

From Existing Results¶

Generate reports from previously saved results:

# Generate specific formats
dotnet run -- report --input ./Results/latest.json --format html,pdf

# Generate all formats
dotnet run -- report --input ./Results/baseline.json --format html,pdf,markdown,console

# Custom output directory
dotnet run -- report --input ./Results/latest.json --format html --output /path/to/reports

JSON Format¶

Overview¶

Machine-readable format containing complete benchmark results with all metrics and metadata.

File Structure¶

{
  "timestamp": "2025-01-18T14:30:22.123Z",
  "machineContext": {
    "machineId": "DEV-MACHINE-01",
    "cpu": {
      "model": "Intel Core i7-9700K",
      "cores": 8,
      "logicalProcessors": 8,
      "maxClockGHz": 3.6
    },
    "memory": {
      "totalMemoryMB": 16384,
      "availableMemoryMB": 8192
    },
    "disk": {
      "totalSpaceGB": 512,
      "freeSpaceGB": 256
    },
    "os": {
      "platform": "Windows",
      "version": "10.0.19045",
      "architecture": "X64"
    },
    "runtime": {
      "dotnetVersion": "9.0.0",
      "isServerGC": true
    },
    "database": {
      "version": "MariaDB 11.7",
      "latencyMs": 23.5
    }
  },
  "summary": {
    "totalBenchmarks": 72,
    "excellent": 65,
    "good": 7,
    "warning": 0,
    "critical": 0,
    "averageMs": 145.3,
    "medianMs": 89.5
  },
  "connectivityResults": [ ... ],
  "repositoryResults": [ ... ],
  "serviceResults": [ ... ],
  "scenarioResults": [ ... ]
}

Use Cases¶

CI/CD Automation: Parse results in build pipelines
Programmatic Analysis: Custom analysis scripts
Data Archival: Long-term storage of benchmark data
Comparison: Input for comparison commands
Baseline Storage: Baseline file format

Configuration¶

{
  "Benchmark": {
    "Reports": {
      "DefaultFormats": [ "json" ]
    }
  }
}

Examples¶

Parse JSON in CI/CD (PowerShell):

$results = Get-Content ./Results/latest.json | ConvertFrom-Json

if ($results.summary.critical -gt 0) {
    Write-Error "Performance critical issues detected!"
    exit 1
}

Write-Host "Performance check passed: $($results.summary.excellent) excellent, $($results.summary.good) good"

Parse JSON in CI/CD (Bash):

#!/bin/bash
critical=$(jq '.summary.critical' ./Results/latest.json)

if [ "$critical" -gt 0 ]; then
    echo "Performance critical issues detected!"
    exit 1
fi

echo "Performance check passed"

HTML Format¶

Overview¶

Interactive web report with charts, tables, and detailed metrics. Optimized for viewing in browsers.

Features¶

Executive Summary: High-level overview with status indicators
Machine Context: Hardware, OS, runtime, database information
Performance Charts: Visual representation of benchmark results
Detailed Tables: Sortable, filterable benchmark data
Latency Analysis: P50, P95, P99 percentiles (REST API suite)
Payload Analysis: Response size distribution (REST API suite)
Interactive Filtering: Filter by status, category, threshold
Dark/Light Theme: Configurable color scheme

Configuration¶

{
  "Benchmark": {
    "Reports": {
      "Html": {
        "IncludeCharts": true,
        "Theme": "dark",
        "Title": "Essert.MF Performance Benchmark Report"
      }
    }
  }
}

Property	Type	Default	Description
`IncludeCharts`	bool	`true`	Include performance charts
`Theme`	string	`"dark"`	Color theme (`"dark"` or `"light"`)
`Title`	string	`"Essert.MF..."`	Report page title

Report Sections¶

1. Executive Summary¶

Performance Benchmark Report
Essert.MF - Standard Suite

Timestamp: 2025-01-18 14:30:22
Machine: DEV-MACHINE-01

Overall Status: EXCELLENT
✓ 65 Excellent | ✓ 7 Good | ⚠ 0 Warning | ✗ 0 Critical

2. Machine Context¶

Hardware & Environment

CPU:     Intel Core i7-9700K @ 3.6 GHz (8 cores)
Memory:  16,384 MB (8,192 MB available)
OS:      Windows 10.0.19045 (X64)
Runtime: .NET 9.0.0 (Server GC)
Database: MariaDB 11.7 (Latency: 23.5ms)

3. Performance Charts¶

Status Distribution (pie chart): Excellent/Good/Warning/Critical
Performance by Category (bar chart): Average times per suite
Latency Percentiles (line chart): P50/P95/P99 over time
Payload Size Distribution (scatter plot): Response size vs. time

4. Benchmark Results Table¶

Benchmark	Category	Avg Time	Status	Threshold	Memory
Product Create	Repository	156ms	Excellent	500ms	2.3 MB
WPC Search	Repository	312ms	Good	500ms	5.1 MB

5. Detailed Metrics¶

For each benchmark: - Average, median, min, max execution time - Standard deviation - Memory allocation - Iterations count - Status and threshold

Use Cases¶

Interactive Analysis: Explore results in browser
Presentations: Share with stakeholders
Documentation: Embed in wiki or documentation
Performance Reviews: Discuss in team meetings
Archival: Visual record of performance over time

Opening HTML Reports¶

# Windows
start ./Results/MACHINE-20250118-143022.html

# macOS
open ./Results/MACHINE-20250118-143022.html

# Linux
xdg-open ./Results/MACHINE-20250118-143022.html

PDF Format¶

Overview¶

Printable report suitable for documentation, presentations, and archival. Generated using QuestPDF library.

Features¶

Professional Layout: Clean, structured page design
Multiple Pages: Automatically paginated
Charts and Graphs: Embedded performance visualizations
Detailed Tables: Comprehensive benchmark data
Machine Context: Full system information
Page Headers/Footers: Title, page numbers, timestamp
Print-Optimized: Optimized for A4/Letter printing

Configuration¶

{
  "Benchmark": {
    "Reports": {
      "Pdf": {
        "IncludeDetailedTimings": true,
        "IncludeCharts": true,
        "PageSize": "A4",
        "Orientation": "Portrait"
      }
    }
  }
}

Property	Type	Default	Description
`IncludeDetailedTimings`	bool	`true`	Include detailed timing tables
`IncludeCharts`	bool	`true`	Include charts (increases file size)
`PageSize`	string	`"A4"`	Page size (`"A4"`, `"Letter"`, `"Legal"`)
`Orientation`	string	`"Portrait"`	Page orientation (`"Portrait"`, `"Landscape"`)

Report Structure¶

Page 1: Cover Page¶

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

      PERFORMANCE BENCHMARK REPORT
              Essert.MF

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Suite:      Standard Suite
Timestamp:  2025-01-18 14:30:22
Machine:    DEV-MACHINE-01

Generated by: Performance Benchmark CLI v1.0.0

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Page 2: Executive Summary¶

Overall status with visual indicators
Performance summary statistics
Machine context overview
Key findings

Page 3+: Detailed Results¶

Results by category (Connectivity, Repository, Service, Scenario)
Detailed timing tables
Performance charts
Recommendations

Last Page: Appendix¶

Full machine specifications
Configuration settings used
Glossary of terms
Contact information

Use Cases¶

Documentation: Include in technical documentation
Presentations: Print for meetings and reviews
Archival: Long-term record keeping
Reports: Management and stakeholder reports
Auditing: Performance audit trail

Tips¶

Landscape Orientation for Wide Tables:

{
  "Pdf": {
    "Orientation": "Landscape"
  }
}

Reduce File Size (Disable Charts):

{
  "Pdf": {
    "IncludeCharts": false
  }
}

Markdown Format¶

Overview¶

Documentation-friendly format ideal for GitHub, GitLab, wikis, and static site generators.

Features¶

GitHub-Flavored Markdown: Compatible with GitHub, GitLab
Summary Tables: Condensed performance overview
Collapsible Sections: Detailed results in expandable sections
Code Blocks: Machine context in formatted code
Status Badges: Visual status indicators
Links: Internal links to sections
Copy-Paste Ready: Easy to embed in documentation

Configuration¶

{
  "Benchmark": {
    "Reports": {
      "Markdown": {
        "IncludeSummaryTable": true,
        "IncludeDetailedResults": false,
        "MaxTableRows": 20
      }
    }
  }
}

Property	Type	Default	Description
`IncludeSummaryTable`	bool	`true`	Include summary table
`IncludeDetailedResults`	bool	`false`	Include all benchmark details
`MaxTableRows`	int	`20`	Maximum rows in tables

Report Structure¶

# Performance Benchmark Report

**Suite:** Standard Suite
**Timestamp:** 2025-01-18 14:30:22
**Machine:** DEV-MACHINE-01

## Summary

| Metric | Value |
|--------|-------|
| Total Benchmarks | 72 |
| ✅ Excellent | 65 |
| ✅ Good | 7 |
| ⚠️ Warning | 0 |
| ❌ Critical | 0 |
| Average Time | 145.3ms |

## Results by Category

### Connectivity Suite (7 benchmarks)

| Benchmark | Avg Time | Status |
|-----------|----------|--------|
| ProcessDb Connection | 23ms | ✅ Excellent |
| StatisticsDb Connection | 28ms | ✅ Excellent |

<details>
<summary>Detailed Results</summary>

#### ProcessDb Connection
- **Average:** 23ms
- **Median:** 22ms
- **Min/Max:** 18ms / 29ms
- **Threshold:** 100ms
- **Memory:** 0.5 MB

</details>

### Repository Suite (45 benchmarks)

...

## Machine Context

```json
{
  "machineId": "DEV-MACHINE-01",
  "cpu": "Intel Core i7-9700K @ 3.6 GHz (8 cores)",
  "memory": "16,384 MB",
  "os": "Windows 10.0.19045 (X64)"
}

Recommendations¶

✅ All benchmarks within acceptable thresholds

💡 Consider optimizing WPC search queries

### Use Cases

- **GitHub README**: Include in repository README
- **Pull Request Comments**: Post results in PR descriptions
- **Wiki Pages**: Add to project wiki
- **Documentation Sites**: Embed in MkDocs, Docusaurus, etc.
- **Issue Tracking**: Reference in performance issues

### GitHub Integration

**PR Comment with Markdown Report:**

```yaml
- name: Post Performance Results to PR
  uses: actions/github-script@v6
  with:
    script: |
      const fs = require('fs');
      const markdown = fs.readFileSync('./Results/latest.md', 'utf8');

      github.rest.issues.createComment({
        issue_number: context.issue.number,
        owner: context.repo.owner,
        repo: context.repo.repo,
        body: `## Performance Benchmark Results\n\n${markdown}`
      });

Console Format¶

Overview¶

Rich terminal output for interactive sessions. Displays results with colors, progress bars, and formatted tables using Spectre.Console.

Features¶

Color-Coded Status: Green (excellent), Yellow (warning), Red (critical)
Progress Bars: Visual progress during benchmark execution
Formatted Tables: ASCII-art tables with borders
Live Updates: Real-time updates during execution
Summary Statistics: Condensed summary at end
No Files Generated: Output to terminal only

Configuration¶

Console format is controlled by command-line flags rather than configuration file:

# Verbose console output
dotnet run -- run --verbose

# Minimal console output
dotnet run -- run --quiet

# Disable colors
dotnet run -- run --no-color

Output Examples¶

Progress Indicator¶

Running Standard Benchmarks...

[████████████████████░░░░░░░░] 75% (54/72)
Current: WPC Search Complex Query (312ms)

Summary Table¶

┌─────────────┬───────┬────────────────┐
│ Category    │ Count │ Status         │
├─────────────┼───────┼────────────────┤
│ Connectivity│ 7     │ 7E 0G 0W 0C    │
│ Service     │ 12    │ 10E 2G 0W 0C   │
│ Repository  │ 45    │ 40E 5G 0W 0C   │
│ Scenario    │ 8     │ 8E 0G 0W 0C    │
└─────────────┴───────┴────────────────┘

Detailed Output (--verbose)¶

[14:30:22] Starting benchmark: Product Create with CRC
[14:30:22]   Iteration 1/10: 158ms
[14:30:22]   Iteration 2/10: 152ms
[14:30:23]   Iteration 3/10: 149ms
...
[14:30:25]   Average: 156ms [Excellent]
[14:30:25]   Memory: 2.3 MB allocated

Use Cases¶

Development: Quick feedback during development
Interactive Testing: Run benchmarks and view results immediately
Debugging: Verbose output for troubleshooting
Quick Checks: Verify changes without generating reports

Format Comparison¶

Feature	JSON	HTML	PDF	Markdown	Console
Machine-Readable	✅	❌	❌	⚠️	❌
Human-Readable	⚠️	✅	✅	✅	✅
Interactive	❌	✅	❌	❌	✅
Charts/Graphs	❌	✅	✅	❌	❌
Printable	❌	⚠️	✅	❌	❌
Version Control	✅	❌	❌	✅	❌
CI/CD Friendly	✅	⚠️	❌	✅	⚠️
File Size	Small	Medium	Large	Small	N/A
Generation Speed	Fast	Medium	Slow	Fast	Fast

Recommendations by Use Case¶

Use Case	Recommended Format(s)
Development	Console
CI/CD	JSON, Markdown
Documentation	Markdown, HTML, PDF
Presentations	PDF, HTML
Archival	JSON, PDF
Analysis	JSON, HTML
Stakeholder Reports	PDF, HTML
PR Comments	Markdown
Interactive Review	HTML, Console

Report Generation Examples¶

Development Workflow¶

# Quick console output
dotnet run -- run --format console

# If results are good, generate full reports
dotnet run -- report --input ./Results/latest.json --format html,markdown

CI/CD Workflow¶

# Generate JSON and Markdown
dotnet run -- run --format json,markdown

# Parse JSON for regression detection
# Post Markdown to PR

Production Baseline Workflow¶

# Generate all formats for comprehensive baseline
dotnet run -- run --iterations 50 --format json,html,pdf,markdown

# Archive all formats

Investigation Workflow¶

# Generate interactive HTML report
dotnet run -- report --input ./Results/slow-run.json --format html

# Open in browser for detailed analysis
start ./Results/slow-run.html

Best Practices¶

File Organization¶

./Results/
├── baseline.json                    # Current baseline (JSON only)
├── 2025-01-18/                     # Daily directory
│   ├── MACHINE-143022.json         # Raw results
│   ├── MACHINE-143022.html         # Interactive report
│   ├── MACHINE-143022.pdf          # Printable report
│   └── MACHINE-143022.md           # Documentation
└── archive/                         # Old results
    └── 2024-12/

Format Selection Guidelines¶

Always Generate JSON
Required for comparisons
Baseline format
Smallest file size
Generate HTML for Interactive Analysis
Best for detailed investigation
Share with team members
Quick visual review
Generate PDF for Formal Documentation
Management reports
Performance audits
Archival records
Generate Markdown for Git Repositories
Commit to repository
Include in PR descriptions
Wiki documentation
Use Console for Quick Feedback
Development iterations
Quick validation
Debugging

Performance Considerations¶

JSON: < 1 second (1-5 MB)
Markdown: < 2 seconds (100-500 KB)
HTML: 2-5 seconds (500 KB - 2 MB)
PDF: 5-15 seconds (1-5 MB)
Console: Real-time output

Generate multiple formats in priority order:

# Fast formats first
dotnet run -- run --format json,markdown,console

# Then generate slow formats if needed
dotnet run -- report --input ./Results/latest.json --format html,pdf

Troubleshooting¶

"PDF generation failed"¶

Solution: Accept QuestPDF license on first run

dotnet run -- report --input ./Results/latest.json --format pdf
# Follow QuestPDF license prompt

"HTML report won't open"¶

Solution: Check file path and browser associations

# Windows: Use full path
start "C:\path\to\report.html"

# Or open in specific browser
"C:\Program Files\Google\Chrome\Application\chrome.exe" report.html

"Markdown not rendering correctly on GitHub"¶

Solution: Ensure GitHub-flavored markdown is enabled

{
  "Markdown": {
    "IncludeSummaryTable": true,
    "MaxTableRows": 50  // GitHub supports large tables
  }
}

"Report file size too large"¶

Solution: Disable charts or reduce detail level

{
  "Html": { "IncludeCharts": false },
  "Pdf": { "IncludeCharts": false },
  "Markdown": { "IncludeDetailedResults": false }
}

Main README - Overview and quick start
Configuration Guide - Configure report formats
Command Reference - Generate reports
Benchmark Suites - What gets reported

Performance Benchmark CLI - Report Formats¶

Overview¶

Available Report Formats¶

Generating Reports¶

During Benchmark Execution¶

From Existing Results¶

JSON Format¶

Overview¶

File Structure¶

Use Cases¶

Configuration¶

Examples¶

HTML Format¶

Overview¶

Features¶

Configuration¶

Report Sections¶

1. Executive Summary¶

2. Machine Context¶

3. Performance Charts¶

4. Benchmark Results Table¶

5. Detailed Metrics¶

Use Cases¶

Opening HTML Reports¶

PDF Format¶

Overview¶

Features¶

Configuration¶

Report Structure¶

Page 1: Cover Page¶

Page 2: Executive Summary¶

Page 3+: Detailed Results¶

Last Page: Appendix¶

Use Cases¶

Tips¶

Markdown Format¶

Overview¶

Features¶

Configuration¶

Report Structure¶

Recommendations¶

Console Format¶

Overview¶

Features¶

Configuration¶

Output Examples¶

Progress Indicator¶

Summary Table¶

Detailed Output (--verbose)¶

Use Cases¶

Format Comparison¶

Recommendations by Use Case¶

Report Generation Examples¶

Development Workflow¶

CI/CD Workflow¶

Production Baseline Workflow¶

Investigation Workflow¶

Best Practices¶

File Organization¶

Format Selection Guidelines¶

Performance Considerations¶

Troubleshooting¶

"PDF generation failed"¶

"HTML report won't open"¶

"Markdown not rendering correctly on GitHub"¶

"Report file size too large"¶

Related Documentation¶