0

I could not find anything related.

And this is strange to me, because it seems like it builds packages for Haddock generation.

uhbif19
  • 3,139
  • 3
  • 26
  • 48

2 Answers2

1

Hackage is a source code repository only, not a binary cache. One of Haskell's biggest weaknesses is a lack of good binary caching. There are nix enthusiasts, some stack solution, and cabal-cache. It is all rather bespoke an problematic. This is in part because GHC does not have a platform independent IR, but I believe that only scratches the surface.

Thomas M. DuBuisson
  • 64,245
  • 7
  • 109
  • 166
  • The main problem is that there are basically infinitely many combinations. GHC does inline code across package boundaries so you would have to cache all packages together with their peer dependencies and also do this for all possible combinations of versions of transitive combinations. Another problem is that you have to compile with all possible compiler versions and compiler flags. This is speculation but maybe it would be possible to cache one stackage version because the compiler version and versions of transitive dependencies are fixed. – Aaronmacaron Jul 25 '23 at 17:53
  • Yes, but those "main problems" are a consequence of the design - or rather a symptom I'd argue. One alternative design that could solve the issues is a compiler that produces a stable IR and keeps compilation artifacts independent of dependency artifacts, deferring optimisms like inlining till link time. – Thomas M. DuBuisson Jul 27 '23 at 13:25
1

Processing the source code to pull out the Haddock markup and produce a nice HTML page is a very different process than actually compiling the code to a binary library.

Massively complicating the idea that Hackage could provide a binary cache is that there is simply no such thing as a single binary artefact that is the compiled version of a package. What you get in the compiled binaries depends not just on the source package, but also the platform you're building it on, the exact compiler version, many compiler flags affect the output, and the exact version of every dependency (including transitive dependencies), etc. There are a huge number of possible variations of every package that would need to be built, if we want any package hosted on Hackage to always be downloadable without building.

In particular, if you were pulling a pre-compiled package from Hackage which also depends on package X, and you also use package X in your own source code (or use other packages that also use X), then all of those usages of package X really need to be the exact same version of package X. There will be many different build plans calling for different binary artefacts corresponding to package X. Covering all of them would be a huge ask, and identifying the "right" ones to pre-build to cover many usage scenarios isn't trivial.

So Hackage doesn't even attempt this. It just provides source code and does documentation builds. Inter-package links are a thing, so it's theoretically also true that there are a huge number of possible HTML pages corresponding to the documentation of package X (depending on which exact versions of other packages it links to). But they're not really that critical to match exactly what you're using in your build; the docs are only for human information purposes, and you can always browse different versions of the linked packages manually if you need to.

Stackage providing binary caches would be simpler, since the whole concept of Stackage snapshots is to fix a single version of every package in the snapshot (all mutually compatible, ideally). That doesn't eliminate all of the difficulties, but does remove a huge chunk of them. I don't believe there is any straightforward binary cache of Stackage snapshots either, though.

Ben
  • 68,572
  • 20
  • 126
  • 174
  • > Beyond the parser, the two have almost nothing in common. Why? Haddock definitely uses hi/hie files now, and it has type-hints, so it probably should do at least type-checking. And there are build logs on Hackage, so it probably already uses hie-cradle or like. > if we want any package hosted on Hackage to always be downloadable without building Yes, I know this is a hard task. My question is not about it, I am just making sure that I did not miss anything in docs/API. Also this task could be solved at least partially, so for me it is strange that Hackage does not do it at all. – uhbif19 Jul 05 '23 at 09:27
  • @uhbif19 You're right, I've removed that "beyond the parser" statement; that was too strong. Nevertheless, I don't find it at all strange that Hackage doesn't attempt to be a comprehensive binary cache. I think you might be underestimating just how many builds of every single package it would have to make to do that (or overestimating how likely it is that a random build plan would hit the cache much if all Hackage did was build each package version once against the latest deps available when it was uploaded, as it does with docs builds). – Ben Jul 05 '23 at 18:02