Skip to content

Commit e3fab03

Browse files
hfytrhfytr
authored
Initial PR (#1)
* Copied from hfytr/PowerParser.jl (personal github) * modified .gitignore * fixed tests, renamed PowerParser -> ExaPowerIO * fixed testss * isbits as test vs assert * parse_matpower is type_stable * no regexes * intermediate timings, fix length(::WordedString) * docs * fix isbits tests * deploy docs * fix docs * docs deploy changes * remove extra fields of Data{T} * test against whole pglib + fixes * fix out type * docs site name * use tap / shift / g_fr / g_to / b_fr / b_to. properly test flips. * fixes. still using col_inds * fix ci * self-hosted is own job * no col inds * optimizations * no items vector * try to fix ci * no word vector. number of allocs is constant wrt to input size * optimizations * reworked benchmarks. results: [ Info: Running ExaPowerIO: 1.327 s (86402 allocations: 809.27 MiB) [ Info: Running JLD2: 61.522 ms (460398 allocations: 174.76 MiB) [ Info: Running PowerModels: 36.598 s (531914384 allocations: 18.62 GiB) * optimizations: [ Info: Running ExaPowerIO: 1.066 s (80752 allocations: 808.78 MiB) [ Info: Running JLD2: 57.796 ms (460398 allocations: 174.76 MiB) [ Info: Running PowerModels: 34.942 s (531914384 allocations: 18.62 GiB) * easy feedback * BusData{T}[] * fixes * iter_to_ntuple takes types and parses * cost terms standardized in parsing * use V directly. change bus_map to ::Vector
1 parent fb51f5a commit e3fab03

File tree

16 files changed

+1146
-1
lines changed

16 files changed

+1146
-1
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
name: CI
2+
3+
on:
4+
pull_request:
5+
branches:
6+
- main
7+
8+
permissions:
9+
actions: write
10+
contents: read
11+
12+
jobs:
13+
test:
14+
runs-on: ${{ matrix.os }}
15+
strategy:
16+
matrix:
17+
os: [ubuntu-latest, windows-latest, macos-latest]
18+
julia-version: ['1.11']
19+
steps:
20+
- name: Checkout repository
21+
uses: actions/checkout@v4
22+
23+
- name: Set up Julia
24+
uses: julia-actions/install-juliaup@v2
25+
with:
26+
channel: ${{ matrix.julia-version }}
27+
28+
- name: Cache Julia artifacts
29+
uses: actions/cache@v3
30+
with:
31+
path: ~/.julia/artifacts
32+
key: artifacts-${{ runner.os }}-${{ hashFiles('**/Project.toml') }}-${{ hashFiles('**/Manifest.toml') }}
33+
restore-keys: artifacts-${{ runner.os }}-
34+
35+
- name: Cache Julia packages
36+
uses: actions/cache@v3
37+
with:
38+
path: ~/.julia/compiled
39+
key: compiled-${{ runner.os }}-${{ hashFiles('**/Project.toml') }}-${{ hashFiles('**/Manifest.toml') }}
40+
restore-keys: compiled-${{ runner.os }}-
41+
42+
- name: Install package dependencies
43+
run: |
44+
julia --project=@. -e 'using Pkg; Pkg.instantiate(); Pkg.precompile()'
45+
46+
- name: Run tests
47+
run: |
48+
julia --project=. -e 'using Pkg; Pkg.test()'
49+
50+
test-self-hosted:
51+
runs-on: self-hosted
52+
strategy:
53+
matrix:
54+
julia-version: ['1.11']
55+
steps:
56+
- name: Checkout Repository
57+
uses: actions/checkout@v4
58+
59+
- name: Set up Julia
60+
uses: julia-actions/install-juliaup@v2
61+
with:
62+
channel: ${{ matrix.julia-version }}
63+
64+
- name: Cache Julia artifacts
65+
uses: actions/cache@v3
66+
with:
67+
path: ~/.julia/artifacts
68+
key: artifacts-${{ runner.os }}-${{ hashFiles('**/Project.toml') }}-${{ hashFiles('**/Manifest.toml') }}
69+
restore-keys: artifacts-${{ runner.os }}-
70+
71+
- name: Cache Julia packages
72+
uses: actions/cache@v3
73+
with:
74+
path: ~/.julia/compiled
75+
key: compiled-${{ runner.os }}-${{ hashFiles('**/Project.toml') }}-${{ hashFiles('**/Manifest.toml') }}
76+
restore-keys: compiled-${{ runner.os }}-
77+
78+
- name: Install package dependencies
79+
run: |
80+
julia --project=@. -e 'using Pkg; Pkg.instantiate(); Pkg.precompile()'
81+
82+
- name: Run tests
83+
run: |
84+
julia --project=. -e 'using Pkg; Pkg.test()'

‎.github/workflows/docs.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
name: Documentation
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
tags: '*'
8+
pull_request:
9+
10+
jobs:
11+
build:
12+
permissions:
13+
actions: write
14+
contents: write
15+
pull-requests: read
16+
statuses: write
17+
runs-on: ubuntu-latest
18+
steps:
19+
- uses: actions/checkout@v2
20+
- uses: julia-actions/setup-julia@latest
21+
with:
22+
version: '1'
23+
- name: Install dependencies
24+
run: julia --project=docs/ -e 'using Pkg; Pkg.Registry.update(); Pkg.develop(PackageSpec(path=pwd())); Pkg.instantiate()'
25+
- name: Build and deploy
26+
run: julia --project=docs/ docs/make.jl
27+
env:
28+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
29+
DOCUMENTER_KEY: ${{ secrets.DOCUMENTER_KEY }}

‎.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,3 +26,9 @@ Manifest*.toml
2626
# File generated by the Preferences package to store local preferences
2727
LocalPreferences.toml
2828
JuliaLocalPreferences.toml
29+
30+
data/
31+
# profiling files generated by julia --track-allocation
32+
*.mem
33+
# generated by PProf.jl
34+
benchmark/profile.pb.gz

‎Project.toml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
name = "ExaPowerIO"
2+
uuid = "14903efe-9500-4d7f-a589-7ab7e15da6de"
3+
authors = ["Archim Jhunjhunwala <[email protected]>"]
4+
version = "0.1.0"
5+
6+
[deps]
7+
PGLib = "07a8691f-3d11-4330-951b-3c50f98338be"
8+
9+
[compat]
10+
PGLib = "0.2.1"
11+
PowerModels = "~0.21"
12+
Test = "~1.11"
13+
14+
[extras]
15+
BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
16+
PowerModels = "c36e90e8-916a-50a6-bd94-075b64ef4655"
17+
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

‎README.md‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,26 @@
1-
# ExaPowerIO.jl
1+
# ExaPowerIO.jl
2+
3+
ExaPowerIO is a minimal IO library for the [Matpower](https://matpower.app/manual/matpower/DataFileFormat.html) file format.
4+
5+
![CI](https://github.com/MadNLP/ExaPowerIO.jl/actions/workflows/ci.yml/badge.svg)
6+
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
7+
[![doc](https://img.shields.io/badge/docs-stable-blue.svg)](https://madsuite.org/ExaPowerIO.jl/stable)
8+
[![doc](https://img.shields.io/badge/docs-dev-blue.svg)](https://madsuite.org/ExaPowerIO.jl/dev)
9+
10+
DICLAIMER: ExaPowerIO is in active development. There may be missing features, documentation, or other issues.
11+
If you experience any of these, please open [Issues](https://github.com/MadNLP/ExaPowerIO.jl/issues), or [Pull Requests](https://github.com/MadNLP/ExaPowerIO.jl/pulls)
12+
13+
### Usage
14+
If you are interested in trying ExaPowerIO, please see the [Usage Documentation](https://madsuite.org/ExaPowerIO.jl/stable/user/) for information on using the functions / structs exported by ExaPowerIO.
15+
16+
### Contributing
17+
If you wish to contribute to ExaPowerIO, please see the [Developer Documentation](https://madsuite.org/ExaPowerIO.jl/stable/dev/) for information on project structure, as well as running benchmarks / tests.
18+
19+
### Alternatives
20+
The main alternative to ExaPowerIO is [PowerModels.jl](https://github.com/lanl-ansi/PowerModels.jl), a monolithic repository which:
21+
- parses and processes Matpower as well as PTI files
22+
- exports formulations for various ACOPF problems
23+
- includes tools to help solve these problems.
24+
Developers wishing to utilize only 1 or 2 of these utilities may find PowerModels bloated.
25+
26+
ExaPowerIO.jl due to its focused nature has superior performance to PowerModels. As seen, ExaPowerIO outperforms PowerModels by a factor of almost 2 in both allocation and timing.

‎benchmark/Project.toml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
[deps]
2+
BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
3+
ExaPowerIO = "14903efe-9500-4d7f-a589-7ab7e15da6de"
4+
JLD2 = "033835bb-8acc-5ee8-8aae-3f567f8a3819"
5+
PGLib = "07a8691f-3d11-4330-951b-3c50f98338be"
6+
PProf = "e4faabce-9ead-11e9-39d9-4379958e3056"
7+
PowerModels = "c36e90e8-916a-50a6-bd94-075b64ef4655"
8+
9+
[compat]
10+
BenchmarkTools = "~1.6"
11+
PGLib = "0.2.1"
12+
PowerModels = "~0.21"

‎benchmark/runbenchmarks.jl‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
using ExaPowerIO, BenchmarkTools, PowerModels, PGLib, Profile, PProf, Logging, JLD2
2+
3+
PowerModels.silence()
4+
5+
NUM_SAMPLES = 10
6+
for (i, arg) in enumerate(ARGS)
7+
if arg == "--num-samples" || arg == "-n"
8+
global NUM_SAMPLES
9+
NUM_SAMPLES = ARGS[i+1]
10+
break
11+
end
12+
end
13+
COMPARE_PM = "--compare-pm" in ARGS
14+
COMPARE_JLD2 = "--compare-jld2" in ARGS
15+
PROFILE = "--profile" in ARGS
16+
CASES = PGLib.find_pglib_case("")
17+
@info """
18+
Running benchmarks with settings:
19+
num-samples: $NUM_SAMPLES
20+
compare to PowerModels: $COMPARE_PM
21+
compare to JLD2: $COMPARE_JLD2
22+
output pprof file: $(PROFILE)
23+
"""
24+
25+
datadir = "../data/"
26+
27+
function run_exapower!()
28+
data = Vector{NamedTuple}(undef, length(CASES))
29+
for (i, dataset) in enumerate(CASES)
30+
data[i] = ExaPowerIO.parse_pglib(Float64, Vector, dataset; out_type=NamedTuple)
31+
end
32+
data
33+
end
34+
35+
function run_jld2()
36+
for dataset in CASES
37+
JLD2.load(joinpath(datadir, dataset * ".jld2"))
38+
end
39+
end
40+
41+
function run_pm()
42+
for dataset in CASES
43+
path = joinpath(PGLib.PGLib_opf, dataset * ".m")
44+
pm_output = PowerModels.parse_file(path)
45+
PowerModels.standardize_cost_terms!(pm_output, order = 2)
46+
PowerModels.calc_thermal_limits!(pm_output)
47+
end
48+
end
49+
50+
@info "Precompiling"
51+
global_logger(ConsoleLogger(stderr, Logging.Warn)) # Consistent logging level
52+
53+
data = run_exapower!()
54+
COMPARE_JLD2 && run_jld2()
55+
COMPARE_PM && run_pm()
56+
57+
GC.gc()
58+
Profile.clear()
59+
global_logger(ConsoleLogger(stderr, Logging.Info))
60+
61+
@info "Running ExaPowerIO:"
62+
global_logger(ConsoleLogger(stderr, Logging.Warn))
63+
if PROFILE
64+
@profile run_exapower!()
65+
pprof()
66+
else
67+
@btime run_exapower!()
68+
end
69+
global_logger(ConsoleLogger(stderr, Logging.Info))
70+
71+
if COMPARE_JLD2
72+
for (i, dataset) in enumerate(CASES)
73+
JLD2.save(joinpath(datadir, dataset * ".jld2"), Dict("data" => data[i]))
74+
end
75+
@info "Running JLD2:"
76+
@btime run_jld2()
77+
end
78+
79+
if COMPARE_PM
80+
@info "Running PowerModels:"
81+
@btime run_pm()
82+
end

‎docs/Project.toml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
[deps]
2+
Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
3+
ExaPowerIO = "14903efe-9500-4d7f-a589-7ab7e15da6de"
4+
Literate = "98b081ad-f1c9-55d3-8b20-4c87d4299306"
5+
LiveServer = "16fef848-5104-11e9-1b77-fb7a48bbb589"

‎docs/make.jl‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
using Pkg
2+
Pkg.activate(@__DIR__)
3+
Pkg.develop(path=dirname(@__DIR__))
4+
5+
6+
using Documenter, ExaPowerIO, Literate
7+
8+
const _PAGES = [
9+
"Introduction" => "index.md",
10+
"User Docs" => "user.md",
11+
"Developer Docs" => "dev.md",
12+
]
13+
14+
makedocs(;
15+
sitename = "ExaPowerIO.jl",
16+
modules = [ExaPowerIO],
17+
remotes = nothing,
18+
authors = "Archim Jhunjhunwala",
19+
format = Documenter.HTML(
20+
prettyurls = true,
21+
sidebar_sitename = true,
22+
collapselevel = 1,
23+
repolink = "https://github.com/MadNLP/ExaPowerIO.jl/tree/main",
24+
edit_link = "main"
25+
),
26+
pages = _PAGES,
27+
clean = false
28+
)
29+
30+
deploydocs(repo = "github.com/MadNLP/ExaPowerIO.jl.git"; push_preview = true)

‎docs/src/dev.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Developer Documentation
2+
This documentation covers how to:
3+
1. Run the projects test suite
4+
2. Run the projets benchmarks
5+
3. Update and build documentation of ExaPowerIO
6+
7+
### Test suite
8+
ExaPowerIO's test suite can be run with the following command:
9+
10+
```bash
11+
julia --project=. -e 'using Pkg; Pkg.test("ExaPowerIO")'
12+
```
13+
14+
The tests compare output from parsing with PowerModels.jl. We realize this is suboptimal, and would encourage PR's changing this.
15+
16+
### Benchmarks
17+
ExaPowerIO's benchmarking suite can be run with the following command:
18+
19+
```bash
20+
julia --project=benchmark benchmark/runbenchmarks.jl
21+
```
22+
23+
Additionally, there are three available flags which can be passed to the benchmarking script:
24+
- ```--compare``` can be passed to output a comparison with PowerModels.jl's parser
25+
- ```--intermediate``` can be passed to output the timing of the two stages of the parser (text -> struct, struct -> named tuple if applicable)
26+
- ```--num-samples <k>``` or ```-n <k>``` can be passed to set the number of samples taken of each case in the benchmark. The default value is 10.
27+
28+
### Documentation
29+
ExaPowerIO uses [Documenter.jl](https://github.com/JuliaDocs/Documenter.jl) for its documentation. To build and deploy the documentation locally:
30+
31+
```bash
32+
cd docs
33+
julia --project -e 'include("make.jl"); using LiveServer; serve(dir="build")'
34+
```
35+
36+
The documentation should then be visible at localhost:8000.

0 commit comments

Comments
 (0)