Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ Available as both a **Rust library** and a **CLI tool**.
- **Swagger 2.0 support** – Swagger 2.0 sources are converted to OpenAPI 3.0.3 automatically; anything that had to be approximated is reported as a warning
- **External `$ref`s** – specs split across files or URLs are bundled per source before merging (CLI sandboxed to the config directory)
- **Validation and CI check** – `--check` validates the merged spec and fails the build when it is invalid or when a committed output file is stale
- **Operation filters** – hide internal endpoints (by path glob, tag, `operationId` or an `x-` flag), globally or per source, with unused components and tags pruned
- **Multiple source types** – local YAML files, local JSON files, and HTTP endpoints (with custom headers)
- **Config-file driven** – define sources and merge options in a single YAML config
- **Conflict resolution** – choose between `error`, `overwrite`, or `rename` strategies for duplicate paths and component names
Expand Down Expand Up @@ -189,6 +190,32 @@ Example GitHub Actions step that keeps a committed spec in sync:

The same validator is available as `openapi_aggregator::validate(&spec)`, and the playground shows its findings in the problems panel.

## Hiding internal endpoints

Build a public spec from sources that also describe internal endpoints by filtering operations out. Filters run **per source, before the merge**, so a hidden operation can never cause a path or component conflict (two services can both expose `/internal/metrics`).

```yaml
merge:
filter: # applies to every source
exclude:
paths: ["/internal/**", "/admin/*"]
tags: [internal]
operation_ids: [debugDump]
extensions: [x-internal] # truthy on the operation or its path item
sources:
- name: billing
path: billing.yaml
filter: # this source only
include:
tags: [public] # allow-list: keep only these operations
```

- **Rules:** an operation matches a rule if *any* of its criteria match (`paths`, `tags`, `operation_ids`, `extensions`). Nothing is hidden by default; list `x-internal` explicitly if you use it.
- **Combining:** `exclude` rules from the global and source level all apply, and `exclude` wins over `include`. `include` is an allow-list; when both levels define one, an operation must satisfy both.
- **Globs:** `*` matches within one path segment (`/admin/*` does not match `/admin/a/b`), `**` matches across segments (`/internal/**`). Everything else, including `{id}`, is literal. Globs apply to `paths` only, not `webhooks`.
- **Cleanup:** a path left with no operations is removed. Components (schemas, parameters, responses, ...), security schemes and tags that were used before filtering and are unused afterwards are pruned. Things that were never referenced to begin with are left alone.
- **Not scrubbed:** the filter does not look inside descriptions, examples, `servers`, `info` or vendor extensions, so check those for internal details yourself. External documents referenced only by hidden operations are still loaded.

## Library Usage

```rust
Expand All @@ -202,6 +229,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
name: Some("petstore".into()),
path: "./specs/petstore.yaml".into(),
additional_blocks: None,
filter: None,
tag_prefix: None,
},
Source::Http {
Expand All @@ -211,6 +239,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
.into_iter()
.collect(),
additional_blocks: None,
filter: None,
tag_prefix: None,
},
],
Expand Down
178 changes: 176 additions & 2 deletions crates/openapi-aggregator-core/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,15 @@ pub struct Config {
impl Config {
/// Parse a YAML (or JSON) config document.
pub fn from_yaml(text: &str) -> Result<Self, crate::error::Error> {
serde_yaml::from_str(text)
.map_err(|e| crate::error::Error::Config(format!("failed to parse config file: {e}")))
let config: Self = serde_yaml::from_str(text).map_err(|e| {
crate::error::Error::Config(format!("failed to parse config file: {e}"))
})?;
if let Some(filter) = &config.merge.filter {
filter.validate("merge.filter").map_err(|e| {
crate::error::Error::Config(format!("failed to parse config file: {e}"))
})?;
}
Ok(config)
}
}

Expand All @@ -43,6 +50,8 @@ pub enum Source {
/// Additional blocks deep-merged into this source spec before merge.
/// Can be used for vendor extensions or any custom OpenAPI blocks.
additional_blocks: Option<Value>,
/// Operations to hide from (or keep of) this source, in addition to `merge.filter`.
filter: Option<FilterConfig>,
},
File {
name: Option<String>,
Expand All @@ -53,6 +62,8 @@ pub enum Source {
/// Additional blocks deep-merged into this source spec before merge.
/// Can be used for vendor extensions or any custom OpenAPI blocks.
additional_blocks: Option<Value>,
/// Operations to hide from (or keep of) this source, in addition to `merge.filter`.
filter: Option<FilterConfig>,
},
}

Expand All @@ -67,6 +78,7 @@ struct RawSource {
headers: Option<HashMap<String, String>>,
tag_prefix: Option<String>,
additional_blocks: Option<Value>,
filter: Option<FilterConfig>,
}

impl TryFrom<RawSource> for Source {
Expand All @@ -78,13 +90,17 @@ impl TryFrom<RawSource> for Source {
.as_deref()
.map(|n| format!("source '{n}'"))
.unwrap_or_else(|| "source".into());
if let Some(filter) = &raw.filter {
filter.validate(&format!("{label} filter"))?;
}
match (raw.url, raw.path) {
(Some(url), None) => Ok(Source::Http {
name: raw.name,
url,
headers: raw.headers.unwrap_or_default(),
tag_prefix: raw.tag_prefix,
additional_blocks: raw.additional_blocks,
filter: raw.filter,
}),
(None, Some(path)) => {
if raw.headers.is_some() {
Expand All @@ -97,6 +113,7 @@ impl TryFrom<RawSource> for Source {
path,
tag_prefix: raw.tag_prefix,
additional_blocks: raw.additional_blocks,
filter: raw.filter,
})
}
(Some(_), Some(_)) => Err(format!(
Expand Down Expand Up @@ -136,6 +153,13 @@ impl Source {
}
}

/// The per-source operation filter, if any.
pub fn filter(&self) -> Option<&FilterConfig> {
match self {
Source::Http { filter, .. } | Source::File { filter, .. } => filter.as_ref(),
}
}

/// Return the tag prefix for this source: custom `tag_prefix` field if set, otherwise the display name.
pub fn tag_prefix(&self) -> String {
match self {
Expand All @@ -146,6 +170,81 @@ impl Source {
}
}

/// Which operations a filter rule applies to. An operation matches when **any** criterion
/// matches. See [`FilterConfig`].
#[derive(Debug, Clone, Default, Deserialize, Serialize, PartialEq, Eq)]
#[serde(deny_unknown_fields)]
pub struct FilterRules {
/// Path globs (`*` within one segment, `**` across segments), e.g. `/internal/**`.
#[serde(default)]
pub paths: Vec<String>,
/// Operations having any of these tags.
#[serde(default)]
pub tags: Vec<String>,
/// Operations with any of these `operationId`s.
#[serde(default)]
pub operation_ids: Vec<String>,
/// Vendor extensions (`x-...`) that are truthy on the operation or its path item.
#[serde(default)]
pub extensions: Vec<String>,
}

impl FilterRules {
fn validate(&self, context: &str) -> Result<(), String> {
if self.paths.is_empty()
&& self.tags.is_empty()
&& self.operation_ids.is_empty()
&& self.extensions.is_empty()
{
return Err(format!(
"{context} must list at least one rule (paths, tags, operation_ids or extensions)"
));
}
for path in &self.paths {
if !path.starts_with('/') {
return Err(format!(
"{context}.paths entry '{path}' must start with '/'"
));
}
}
for extension in &self.extensions {
if !extension.starts_with("x-") {
return Err(format!(
"{context}.extensions entry '{extension}' must start with 'x-'"
));
}
}
for (key, entries) in [("tags", &self.tags), ("operation_ids", &self.operation_ids)] {
if entries.iter().any(|e| e.is_empty()) {
return Err(format!("{context}.{key} entries must not be empty"));
}
}
Ok(())
}
}

/// Select which operations end up in the merged spec: `exclude` hides matching operations,
/// `include` (an allow-list) keeps only matching ones. Exclude wins.
#[derive(Debug, Clone, Default, Deserialize, Serialize, PartialEq, Eq)]
#[serde(deny_unknown_fields)]
pub struct FilterConfig {
pub include: Option<FilterRules>,
pub exclude: Option<FilterRules>,
}

impl FilterConfig {
/// Check the rules are well-formed; `context` names the config location for errors.
pub fn validate(&self, context: &str) -> Result<(), String> {
if let Some(rules) = &self.include {
rules.validate(&format!("{context}.include"))?;
}
if let Some(rules) = &self.exclude {
rules.validate(&format!("{context}.exclude"))?;
}
Ok(())
}
}

/// Output configuration.
#[derive(Debug, Clone, Deserialize, Serialize)]
#[serde(deny_unknown_fields)]
Expand Down Expand Up @@ -198,6 +297,8 @@ pub struct MergeConfig {
/// If set, replaces any tags collected from sources.
/// (Operation-level tag references are NOT rewritten; these should match.)
pub tags: Option<Vec<TagEntry>>,
/// Operations to hide from (or keep of) every source.
pub filter: Option<FilterConfig>,
}

impl Default for MergeConfig {
Expand All @@ -210,6 +311,7 @@ impl Default for MergeConfig {
info: None,
servers: None,
tags: None,
filter: None,
}
}
}
Expand Down Expand Up @@ -334,7 +436,79 @@ mod tests {
headers: HashMap::new(),
tag_prefix: None,
additional_blocks: None,
filter: None,
};
assert_eq!(source.display_name(), "billing.example.com");
}

const FILTER_YAML: &str = "
sources:
- name: billing
path: billing.yaml
filter:
include:
tags: [public]
merge:
filter:
exclude:
paths: ['/internal/**', '/admin/*']
tags: [internal]
operation_ids: [debugDump]
extensions: [x-internal]
";

#[test]
fn parses_filters() {
let config = Config::from_yaml(FILTER_YAML).unwrap();
let global = config.merge.filter.as_ref().unwrap();
assert!(global.include.is_none());
let exclude = global.exclude.as_ref().unwrap();
assert_eq!(exclude.paths, vec!["/internal/**", "/admin/*"]);
assert_eq!(exclude.tags, vec!["internal"]);
assert_eq!(exclude.operation_ids, vec!["debugDump"]);
assert_eq!(exclude.extensions, vec!["x-internal"]);

let source_filter = config.sources[0].filter().unwrap();
assert_eq!(source_filter.include.as_ref().unwrap().tags, vec!["public"]);
assert!(source_filter.exclude.is_none());
}

#[test]
fn configs_without_filters_have_none() {
let config = Config::from_yaml("sources:\n - path: a.yaml\n").unwrap();
assert!(config.merge.filter.is_none());
assert!(config.sources[0].filter().is_none());
let http = Config::from_yaml("sources:\n - url: https://a.test/x\n").unwrap();
assert!(http.sources[0].filter().is_none());
}

#[test]
fn invalid_filters_are_config_errors() {
let global =
|rules: &str| format!("sources:\n - path: a.yaml\nmerge:\n filter:\n {rules}\n");
let source = |rules: &str| {
format!("sources:\n - name: s\n path: a.yaml\n filter:\n {rules}\n")
};
let cases = [
(global("exclude: {tag: [x]}"), "tag"),
(global("include: {}"), "merge.filter.include"),
(global("exclude: {tags: []}"), "at least one rule"),
(
global("exclude: {paths: ['internal/**']}"),
"must start with '/'",
),
(
global("exclude: {extensions: [internal]}"),
"must start with 'x-'",
),
(global("exclude: {tags: ['']}"), "empty"),
(source("include: {}"), "source 's'"),
(source("exclude: {operation_ids: ['']}"), "empty"),
(source("exclude: {paths: ['x']}"), "must start with '/'"),
];
for (yaml, fragment) in cases {
let err = Config::from_yaml(&yaml).unwrap_err().to_string();
assert!(err.contains(fragment), "expected '{fragment}' in: {err}");
}
}
}
Loading
Loading