Migrate from the .cabal format to a widely supported format

In the wake of the exact-printer initiative, I proposed another approach: why not say good bye to the .cabal file format and switch to something that is widely supported.

There is a few alternatives, the most important attribute would be that they are widely supported in industry.

JSON
YAML
TOML

Note that all of these are (mostly) isomorphic to JSON (scalars, lists, dicts), which is important for easy translation between them (e.g. for config generation purposes).

What would this give the Haskell ecosystem?

Editor support: every modern editor like vscode has a way of assigning JSON schema to a file, which gives completion and inline documentation for free everywhere
Also, syntax highlighting and auto formatting come for free
Cabal doesn’t need to implement its own parser. If JSON is chosen, the parser is even context-free.

What would it give to users?

Instant familiarity with the format: you don’t touch cabal files all too often as a user, so you don’t want to learn yet another syntax
Templating cabal config with standard tooling (e.g. jq, yj), which is important e.g. in a monorepo context
Inline documentation without setup

What are others doing?

Most modern package managers that don’t go the full turing-complete configuration route (e.g. Scala’s sbt, Erlang) usually converge their config on a widely supported syntax.

Examples:

npm, yarn (package.json, package-lock.json)
Stack (stack.yaml, stack.yaml.lock)
hpack (project.yaml)
cargo (Cargo.toml)
Elm (elm-package.json)
Maven (pom.xml)
poetry (pyproject.toml, poetry.lock)

Counterexamples:

go (go.mod), though flat shasums and go packages have no configuration file
pip (requirements.txt), though see poetry above
sbt, hex: both use their turing complete parent languages
leiningen (project.clj), clojure is a lisp, and sexps are already a data format

I don’t expect cabal would drop support for the cabal file format very soon, rather it would start out by generating a .cabal file from the .json/.toml/.yaml for consumption by older version of cabal. Then after a multi-year grace period, the new format would become the standard and projects could drop their autogenerated .cabal files.

Note that some people have mentioned dhall as a possible alternative, but using it would destroy most benefits, namely:

Familiarity
Editor support
widely available tooling for ops integration
simple parser
isomorphic to json

However, I expect that there would be a dhall library for generating cabal.json files, which can aid into integrating dhall-based (dev)ops setups with Haskell packages.

Is this cabal.json the plan.json from the cabal docs or is it a .cabal in JSON format?

plan.json (JSON)
A JSON serialization of the computed install plan intended for integrating cabal with external tooling. The cabal-plan package provides a library for parsing plan.json files into a Haskell data structure as well as an example tool showing possible applications.

Note that some people have mentioned dhall as a possible alternative

Also note that this already exists as dhall-to-cabal. While its goal was to generate .cabal files, that's not the only solution. A more integrated solution would be a cabal-install that can actually consume these files. I'm not saying this is the solution, just mentioning this as prior art. I'll step out of the conversation for now and let others share there thoughts, but if any one wants to talk about Dhall in particular here, I do have thoughts

Is this cabal.json the plan.json from the cabal docs or is it a .cabal in JSON format?

It is the current .cabal file in a not-home-grown syntax

I proposed another approach: why not say good bye to the .cabal file format and switch to something that is widely supported.

When I think about it, don't think these are mutually exclusive tickets: an exact printer gets us a reasonable source representation. This is good for a few reasons: we can derive translational tools from the same representation - we only need change the parser and the printer. This frees up efforts to migrate between formats!

There is a few alternatives, the most important attribute would be that they are widely supported in industry.

Of the three suggestions, TOML is the most attractive. YAML is has too much variable syntax, and JSON is aesthetically (and mechanically) displeasing for me to write as a human. TOML's grammar is minimal and admits a small and easy to generate + verify parser and lexer (note: toml-parser is a little outdated) that eliminates the need for us to write it from scratch. In fact, maintaining this would be a dream, and would be a boon for tooling, since we can derive an ABNF for our specific flavor quite easily.

Then after a multi-year grace period, the new format would become the standard and projects could drop their autogenerated .cabal files.

👍

As an experiment, I took a library I wrote a few years ago and manually converted its Cabal file to TOML. I like it! The conversion can be totally systematic.

TOML was noticeably nicer to edit—Emacs has a simple built-in TOML mode and I didn't have to worry about indentation/formatting. (I've done Haskell for over a decade now and I'm still not consistent in how I format Cabal files!) Structured commands for navigating and editing the TOML file would be nice; I don't know if something like this already exists, but if it doesn't, adding it to Emacs would be easy. I wouldn't even think of trying something like that for Cabal's custom syntax.

I've used YAML a lot more than TOML in the past. Compared to YAML, I found needing to quote all my strings a bit annoying; on the other hand, TOML was much nicer to pick up and doesn't have weird corner cases to worry about. At work I recently ran into some weird YAML files that used anchors in a way that didn't work in Python—not something that would happen with TOML.

In my dream world we would use an S-expression based syntax (like sexplib) but I know that is not to be :(.

I immediately found that multiline strings were useful. Multiline strings and comments seems like the bare minimum for a human-oriented format; YAML and TOML support that, JSON doesn't.

It's a bit long, but here's the whole file:

cabal-version = "2.2"

[package]
name = "modular-arithmetic"
version = "2.0.0.1"
synopsis = "A type for integers modulo some constant."
description = """
A convenient type for working with integers module some constant. It saves you from manually wrapping numeric operations all over the place and prevents a range of simple mistakes. @Integer `Mod` 7@ is the type of integers (mod 7) backed by @Integer@.

We also have some cute syntax for these types like @ℤ/7@ for integers modulo 7.
"""
homepage = "https://github.com/TikhonJelvis/modular-arithmetic"
bug-reports = "https://github.com/TikhonJelvis/modular-arithmetic/issues"
license = "BSD-3-Clause"
license-file = "LICENSE"
author = "Tikhon Jelvis <[email protected]>"
maintainer = "Tikhon Jelvis <[email protected]>"
category = "Math"
build-type = "Simple"
extra-source-files = ["README.md", "CHANGELOG.md"]

[source-repository.head]
type = "git"
location = "git://github.com/TikhonJelvis/modular-arithmetic.git"

[library]
hs-source-dirs = ["src"]
ghc-options = ["-Wall"]
default-language = "Haskell2010"
exposed-modules = [
  "Data.Modular"
]
build-depends = [
  "base >4.9 && <5",
  "typelits-witnesses <0.5"
]

[test-suite.examples]
hs-source-dirs = ["test-suite", "src"]
main-is = "DocTest.hs"
default-language = "Haskell2020"
type = "exitcode-stdio-1.0"
build-depends = [
  "base >4.9 && <5",
  "doctest >= 0.9",
  "typelits-witnesses <0.5"
]

Another benefit: the format would be naturally extensible. Cabal could provide a section for plugin/tool/etc config, and tools would have no issues parsing values from there. I'm imagining something like this:

[plugin.liquid-haskell]
smt-solver = "z3mem"

My experience has been that providing "extension points" in formats is always useful. We can't figure out everything people want to do with their libraries ahead of time but we can make the format adaptable. If people need something Cabal doesn't support, they can add it while still keeping a single canonical file for library-specific settings.

For yaml there's also of course hpack. So anyone who wants to write cabal files in either yaml or dhall is welcome to do so. Note that we don't have exactprinters for either of those formats either, as far as I know. As I recall, due to the semantics of yaml, conditional clauses are rather unpleasant there, among a few other issues (and pretty-printing reorders things in unpleasant ways as well). (And also as emily notes, the yaml grammar is rather complicated as is).

Toml does seem promising, but I worry that its support for conditionals or other more complex syntax wouldn't be particularly great either. Translations of some more complex files might be worthwhile, to experiment with this.

Btw, note that cabal is already extensible, via "x-" fields.

In any case, I think the right next step is to get the cabal grammar pinned down and to have an exactprinter for at least the format we already have and is widespread.

@gbaz

This is good for a few reasons: we can derive translational tools from the same representation - we only need change the parser and the printer. This frees up efforts to migrate between formats!

I like this and am the maintainer of the translational tool hpack-dhall that can translate:

dhall -> cabal
dhall -> json
dhall -> yaml # the package.yaml format of hpack
dhall -> dhall # with imports resolved

I am a bit wary about each format being capable of doing a faithful representation. For instance, hpack's conditionals can break dhall's typing. This is the trouble @gbaz just mentioned.

@djc

I want this to work, but unfortunately I see some issues with all the proposed formats so far. I think TOML / YAML / JSON will never work, but Dhall, while it might not be a good fit today, can be made to work.

TOML / YAML / JSON

If Cabal files were merely data they would be fine, but unfortunately they are code, due to the conditionals, and parameters used in those conditions. This is true of Cargo packages too, and the solution there has been to stuff syntax into strings. Firstly, this largely defeats the point as we still need application-specific parsers (and pretty printers!) to handle those strings.

But more worryingly, I have reason to believe this has warped the design process of Cargo. See, for example, the back and for with @djc and me in rust-lang/rfcs#3143, where @djc agrees Cargo has backed itself into a corner, but objects to my further using strings, or trying to encode the information in a more structured but awkward and verbose way. I may have disagreed with @djc on which unpleasant choice too take, but I absolutely do agree that TOML forcing Cargo into this awkward situation is trajic, and no one shouldhave to pick between those unpleasant options in the first place.

Cabal converting from the existing design avoids some of the distortion from TOML's perverse incentives, but I have no doubt the language of Cabal files will continue to evolve, and I don't want "TOML goggles" to mess things up going forward.

Dhalll

Dhall is an actual programming language, and therefore squarely fixes the above issues. And to be clear I would really like to endorse Dhall as it is the right sort of way to make these things conform to a standard. There are two quibbles with Dhall as it currently exists however, that I think should be addressed first:

imports / IO. As far as I know, Dhall always allows downloading arbitrary stuff, etc. as long as you give it a content address of some sort to make it pure (like fixed output derivations). This is a fine design in general, but I worry about e.g. cabal2nix needing internet access to do it's job, which (ironically giving the nix inspiration!) would be a regression and major pain. If there is a way force Dhall programs to be more self-contained, that would assuage my concern.
abstract interpretation. The Dhall model somewhat assumes that dhall will evaluate a closed term, spltting out a value for the consuming application to deal with. But this doesn't totally reflect how Cabal works. Today, we have automatic flags, which means we need to vary parameters based on results. With a few flags this can be brute forced, but with more abstract interpretation is much more efficient. Dhall's strong normalization is good to make such static analysis tractable, but we might also want additional restrictions to make it efficient and also easier for humans to understand.

Maybe that is overkill for manual flags, but @mpickering's and my presentation on what comes after CPP (https://icfp19.sigplan.org/details/hiw-2019-papers/9/Configuration-but-without-CPP, https://www.youtube.com/watch?v=YupkE1vsZ4o) has gotten me thinking about abstract interpretation more broadly. Eventually we want to tackle the goal of "type safe packaging" i.e. ensuring all valid version solving solutions will in fact compile. It's hard, but not tackling it is anathema to our values, and abstract interpretation of various sorts is key to making it work.

So yeah, in conclusion I want Dhall to work, but it's important we we be able to restrict ourselves to a sort of "mini Dhall" so we can do this analysis and we will have to integrate Dhall with Cabal fairly deeply. I'm not sure whether the current Dhall implementation supports such a restricted "mini Dhall", but that can easily be fixed.

Then after a multi-year grace period, the new format would become the standard and projects could drop their autogenerated .cabal files.

Remember that Hackage is an append-only repository. It would be utterly disappointing if a future version of Cabal would be unable to build an old package just because it no longer parses its very own package format. So I don't think it would be wise to abandon a parser of Cabal files even after a very long grace period. And if we are to retain the parser and all its complexity, than what exactly are we to gain? What about other tooling (e. g., Stack)?

With regards to editor support, why aim for a generic JSON autocompletion? These days we should not settle for anything less than a domain-specific language server, and custom format is not a hindrance for it.

I'm sorry if my tone sounds harsh, but I'm afraid we are chasing an ideal to the detriment of compatibility, as it's very customary in Haskell community.

The hpack workflow has already been sufficiently explained to be worse for usability

I don't think so, it's been explained as complicating the tool design. But it would be better for usability.

Likewise, integrating ghcup's automatic GHC installation would be a big improvement for usability, but a complication from a design perspective.

You're on the cabal issue tracker.

Changing the .cabal format is an infrastructure thing that affects both the cabal-install tool and stack equally.

You're on the cabal issue tracker.

Changing the .cabal format is an infrastructure thing that affects both the cabal-install tool and stack equally.

You're moving goalposts.

You told people on the cabal issue tracker that they should use stack if they want usability.

At this point I'm not sure if you're trolling.

Yes, it will affect stack as well, but stack doesn't do its own parsing of cabal files. The Cabal API used can stay largely the same. Pantry might need some adjustment, but that won't be hard.

I'm saying, if you care about cabal-install usability, you should copy features from stack, as some people are asking for (#8605). But this is in tension with your desire to keep the design of cabal-install simple.

You need to pick one: do you want a single tool that does everything (most usable), or do you want cabal-install a tool that just does one thing well (simpler design)?

@angerman

@angerman

And this is a direct result of the compiler breaking existing packages that are perfectly fine.

If you do not upgrade to a new compiler version, you can keep your existing packages just fine.

It tends to be pretty hard to stay behind in practice, for the reasons you indicated after this.

Now almost every compiler release requires material changes to the package.

In any case this is not the correct thread to discuss this.

I think it is. Such local decisions compound and affect the community as a whole. Using this logic, projects would be making decisions about how people are affected by breaking changes considering only their own package.

When you consider that similar thinking is being applied to many other parts of the Haskell ecosystem, then it's easier to see that the instability of it compounds and becomes too much. No single project will be able to completely stop it. It takes all projects together to do so. So it's not just that this conversation is important here, it's that it's important every time it comes up in any project that is core to the community.

As others have said, it's an open source project and everyone is free to spend time on what they deem interesting.

True, although as a community we are all contributing, and we all rely on cabal & ghc do to it. Changes to those packages affect every one of us. Someone being free to spend their time on whatever they want doesn't mean that the change should be accepted in cabal or ghc.

@ivanperez-keera

@ivanperez-keera yes, that was precisely my point. The failure to keep any resemblance of backwards compatibility, and upstream continuously hard breaking the whole ecosystem has massive ripple effects. If maintainers were to spend less time dealing with the churn of adapting to new compiler version that simply reject their code, you could upgrade the compiler, and still use the old library perfectly fine. If you decide to upgrade to a newer major version of that library, that is your choice, and not one force onto you by trying to stay somewhat current with the compiler. Right now new compiler, almost always implies a slew of new dependencies. Why? Because the compiler (with each 6mo release) introduces breaking changes, that make old code incompatible. It's insane!

I think we completely agree on this topic. And while I see the relation to breaking the cabal format, I still would like to not derail this thread too much into that direction.

Yes, it's everyones time and they are free to do what they want with their time, and what hopefully makes them happy. I did not say anything about me supporting such a change in anyway. I'm highly skeptical. I will remain open minded, but I would want to see a purportedly better format to be better in every dimension, not just read/write support for it in some other non-haskell language.

I don't think cabal is perfect. And I absolutely hate the conditional logic in it. It's almost as bad a CPP from a flattening out standpoint. But I will give the cabal format that it's fairly straight forward.

If someone can come up with a better format, that doesn't regress in any significant way, solves lots of warts the cabal format has, I'm happy to lend my support to it; I have not yet seen it though, but that doesn't mean it doesn't exist.

Hey just wanna bring life to this old issue!

I made a little example of how a Cargo style interface could make cabal way easier to use.
https://github.com/matthunz/hoot

We've been talking on reddit https://www.reddit.com/r/haskell/comments/14v3wo7/would_anyone_be_interested_in_hoot_a_cabal/
about how TOML could be helpful

Hey just wanna bring life to this old issue!

I made a little example of how a Cargo style interface could make cabal way easier to use. https://github.com/matthunz/hoot

We've been talking on reddit https://www.reddit.com/r/haskell/comments/14v3wo7/would_anyone_be_interested_in_hoot_a_cabal/ about how TOML could be helpful

I’m just a single person of course, but I’m coming back to Haskel after a few years in Rust and other languages. I strongly disagree with the comments in this issue about this being an unnecessary change. I think I remember one of them being “there isn’t real demand, this is just one person asking for something”

I don’t think that’s true. I think many of us who would benefit from cabal working similarly to other modern build systems are just less likely to be core Haskell contributors. I’m trying to get involved and make up for that now, but I suspect there are many others like me who would benefit.

@seanhess

@seanhess I think nobody is questioning that using a "widely supported format" would be useful, that much is clear. The point is that there are two obstacles:

Migration and backward compatibility
The new format has to be sufficiently expressive and performant
- Migrate from the .cabal format to a widely supported format #7548 (comment)
- Migrate from the .cabal format to a widely supported format #7548 (comment)

Any proposal for a new format would have to address those two concerns, and as far as I can see so far none did.

I don't want to sound condescending, but this ticket is already really long¹ so please read the conversation before commenting, or at least the comments I linked above and/or the ones with most reactions.

119 comments, the longest ticket in this tracker ↩

Thanks for sharing! I read a little into that issue but those comments really shed a lot of light. I do still think that the advanced cabal examples (such as https://hackage.haskell.org/package/raaz-0.3.0/raaz.cabal) could be solved with something like Rust's feature flags

I find myself strongly opposed to the idea of moving to a common format, so here are the counterpoints:

Instant familiarity

There's An INI critique of TOML that argues the precise opposite through the lens of a person without any prior knowledge of the format. Having to learn another syntax is arguably a lesser evil than having to learn another flavor of TOML/YAML.

An extreme case of this is Kubernetes with its box of DevOps joke.
Ease of implementation

This is only true for small configuration files that fully fit into the spec of the format. For this specific case the library that parses/serializes the format already does everything you need it to do.

With an extra format you still require a parser, it's just that instead of parsing from a sequence of bytes or a custom grammar you are now bound to a far wider grammar with some tokens outright missing (build-depends was mentioned many times in this discussion). At best this means you're tying yourself to libraries and/or external tools that process the common format, at worst you just signed up for writing said libraries and/or external tools.
Syntax highlighting

As in syntax highlighting for the wider format, it wouldn't magically know the meaning of the keywords Cabal introduced. If the highlighting is going to be partial or incorrect, everything might as well just be one color.
Templating

I don't get the appeal and I assume the reasons for this are pragmatic, so I feel like templating is only brought in when there is no other way to create or alter the file. Note that format changes up to date have been very conservative: deprecations are rare and no sections have been removed. Some day Cabal will do a major bump that rewires something big and everyone will have to spend several hours upgrading their meta-tooling.

The correct answer here would be a separate tool to do this, ideally maintained by Cabal (call it cabal edit), which would allow flexibility in moving and removing certain options in a backwards-compatible manner.

All in all the gains are speculative, overshadowed by the sheer girth of maintenance a change like this would entail. I fully agree with #7548 (comment), better tooling is both easier to define and far more desirable.

TL;DR, I challenge JSON, ..., Dhall suggestors to model e.g.

https://hackage.haskell.org/package/transformers-compat-0.7/transformers-compat.cabal
https://hackage.haskell.org/package/raaz-0.3.0/raaz.cabal

in their favourite "syntax" format. Otherwise this discussion is just
wasting everyones time by not being concrete.

I dunno... Doesn't seem that bad to me ¯\_(ツ)_/¯
https://gist.github.com/vanceism7/9db49c255b5ca1677e11fdf17a699fda

(I'm being super tongue-in-cheek here, I just pasted the cabal file to chat-gpt and had it generate equivalents for me in json and yaml. I'm sure maybe there's some difficulty here I'm missing, but the yaml file in specific looks pretty neat in my eyes)

some difficulty here

All those duplicated if, then, else keys are likely problems for many implementations of either JSON or YAML, for starters.

Those files won't work for a huge variety of reasons, basically all of which have been discussed at length in this thread. This is why we need people to design protocols through thought and discussion, rather than asking machines to churn out "neat" looking but wrong slop.

This issue has stalled for a while. The benefits of switching currently to a different format are severely outweighed by the cost of 1) unavailable features (e.g., syntax highlighting), 2) implementation effort, 3) temporary introduction of complexity in cabal (until all old formats can be removed, which would take years) and other tools that work with cabal files, 4) potential need for additional tooling, 5) the need for new learning by people in the community, 6) adaptation effort in the Haskell ecosystem (packages would have to update), 7) documentation that would have to be put together and maintained for years while the community transitions. Many of these points, and others, were very well captured by #7548 (comment) and #7548 (comment). Cabal has a long history of breaking the interface, and this would simply add to that and create more breakage of packages, in a community that already struggles to put enough energy to keep packages well maintained.

There are currently 416 bugs open in Cabal's repo (out of 1521 issues total), many going back for many years.

I propose that we simply close this item as "not for now" and focus on stability before moving on to bigger changes that would require a huge investment by the community at large.

General note

I would also like to ask people who propose new changes to take the seat of devil's advocate, and try to think also of good reasons not to do things, as well as the cost that it has for everybody (almost literally: imagine if you had to pay everyone in the community who's going to spend time as a consequence of this change $250/h; how much would that rack up to?), and how it makes the ecosystem better as a whole for everyone (beyond their own specific interest). Many of us are also investing a lot of effort into promoting haskell and getting it adopted into our companies, which would benefit the community at large. Understanding the impact for the client and the user is important to increasing Haskell's adoption.

Navigation Menu

Repository navigation

Migrate from the .cabal format to a widely supported format #7548

What would this give the Haskell ecosystem?

What would it give to users?

What are others doing?

TOML / YAML / JSON

Dhalll

100 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Migrate from the .cabal format to a widely supported format #7548

Description

What would this give the Haskell ecosystem?

What would it give to users?

What are others doing?

Activity

Profpatsch commented on Aug 15, 2021

Profpatsch commented on Aug 15, 2021

philderbeast commented on Aug 15, 2021

ocharles commented on Aug 15, 2021

Profpatsch commented on Aug 15, 2021

emilypi commented on Aug 15, 2021

TikhonJelvis commented on Aug 15, 2021

TikhonJelvis commented on Aug 15, 2021

gbaz commented on Aug 15, 2021

philderbeast commented on Aug 15, 2021

Ericson2314 commented on Aug 15, 2021

TOML / YAML / JSON

Dhalll

Bodigrim commented on Aug 15, 2021

100 remaining items

AshleyYakeley commented on May 26, 2023

AshleyYakeley commented on May 26, 2023

hasufell commented on May 26, 2023

AshleyYakeley commented on May 26, 2023

ivanperez-keera commented on May 26, 2023

angerman commented on May 26, 2023

matthunz commented on Jul 10, 2023

seanhess commented on Jul 10, 2023

fgaz commented on Jul 10, 2023

Footnotes

matthunz commented on Jul 10, 2023

BurningWitness commented on Oct 2, 2023

vanceism7 commented on Oct 18, 2023

rhendric commented on Oct 18, 2023

gbaz commented on Oct 18, 2023

ivanperez-keera commented on Jan 24, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions