0

One of the brilliant aspects of Firely.Terminal is its ability to interoperate with the local FHIR package cache (~/.fhir) in a way that is fully compatible with HAPI tools using the cache. Sadly, that no longer seems to be the case.

Today I updated Firely.Terminal to version 2.4.2 and it seems that the new version walks all over the FHIR package cache, changing files without having been asked to.

It used to be that the only thing Firely.Terminal changed in existing packages was the generation of a missing .index.json. For newly installed packages, the only difference to a HAPI-installed package was some additional fields in .index.json (presence of some fields containing null which would normally be suppressed, and the addition of a fhirVersion field).

When the new Firely.Terminal is told to add a package to a scope (fhir install) it automatically 'bakes' it, which seems to involve things like snapshotting all StructureDefinition resources and expanding all ValueSet resources. Even resources whose content remains unscathed get their timestamps trashed. The same fate befalls all packages that are listed as dependencies in the manifest of the package being added to the scope.

There is an 'unbake' command (e.g. fhir unbake --package kbv.ita.for@1.0.1) but this does not operate recursively. What's more, when it says 'Bake successfully removed from KBV.ITA.FOR@1.0.1' (note the erroneous capitalisation) then that is an outright lie - the contents of the package directory are completely unchanged, except for the removal of the file .bake.json.

Hence the only way of restoring the package cache to working order is to identify all trashed packages, delete them all, and then reference them with some HAPI tool in order to get them re-cached.

I wouldn't mind so much if Firely.Terminal trashed its own cache. But what it destroys is the global HAPI package cache for the current user, and that is simply not acceptable.

Is there any way of suppressing the destructive behaviour of Firely.Terminal? Ideally globally (with machine-wide effect), but a secret command switch would do in a pinch. If that is not possible: does anyone know which of the older versions is the newest that still works, and where to get it?

Note: if the cached packages are write-protected then Firely.Terminal doesn't take the hint - it tries to clobber the files anyway and spews out oodles of 'access denied' messages. What's more, it doesn't even stop when an error occurs; instead it continues on its merry way and trashes everything that one might have forgotten to write-protect.

Background: one of the properties of the FHIR package cache that is important for our work is that the files in the cache are exactly the same as those in the (normative) published packages. In particular, we need profiles published without snapshots to not contain snapshots, value sets published without expansions to not contain expansions and so on. For one thing, this makes it possible to verify that the cached files are exactly the same as those contained in the published packages (or fixed versions thereof). For another, we need to control the context in which profiles are snapshotted, value sets expanded and so on because it may be necessary to supply dependencies that are different from those declared in the package manifest. The latter is sometimes necessary because the profile/package version management in the context of electronic prescriptions in Germany is a bit, erm, peculiar and can diverge from FHIR standards. For this to work at all the resources must be snapshotted/expanded dynamically (depending on the use context), not statically on disk. Things are moving in a more standards-compliant direction but we are not quite there yet.

DarthGizka
  • 4,347
  • 1
  • 24
  • 36

1 Answers1

1

Latest version without bake (on install)

From some quick testing of the latest versions of Firely Terminal, it seems 2.2.0 is the latest without bake functionality (and auto-bake on install). Installation instructions:

> dotnet tool uninstall --global Firely.Terminal
> dotnet tool install --global Firely.Terminal --version 2.2.0

Baking

The bake functionality has been introduced to provide packages with snapshots, because not all downstream tooling (most notably sushi) are able to generate these themselves.

Currently bake might be a little too aggressive by default, also recalculating snapshots for packages that already have them. In principle, this should not be a problem since snapshots are a just a cache for the calculation of all the layered differentials. Since snapshot logic still evolves it might even be desirable now and then to recalculate. But in newer versions we will look to:

  • Change the default to not recalculate when already provided
  • Provide a global setting to change that default to never calculate/always (re)calculate snapshots

This should prevent Firely Terminal from touching any files that don't need touching in the package cache. I'm not sure from your question if there was anything broken in the state of the shared FHIR cache after 'baking', given your use of 'thrashing' and 'destroying'?

Unbaking

The unbake command is intended to remove snapshots from a folder of packages. I see in my testing that it's not doing that, which I'll take as an issue to fix.

  • Thank you, that answers my question and it gives useful background to boot. I've edited my post to take some of the edge off, and I added some background information. – DarthGizka Mar 14 '22 at 15:17
  • However, some critique points remain. The 'bake' functionally would be an unconditionally good thing if it were at the discretion of the user whether to apply it or not. That would allow users of tools like sushi to work around shortcomings in those tools, and it would allow other users to improve startup times for other tools by avoiding the cost of snapshotting/expansion at program startup. However, it should *at least' be possible to suppress the auto-baking. IMHO it would be even better if the default were not to auto-bake, because it changes the semantics of the package cache. – DarthGizka Mar 14 '22 at 15:23
  • One scenario that requires users to have absolute control over their package cache (and in particular, attempts to change the contents of the cache) is off-line use, where a well-curated package cache is implanted in a computer without Internet access (e.g. in a health care context). That allows some use of standard tools that need the package cache, despite the missing Internet connection. – DarthGizka Mar 14 '22 at 15:30
  • Another usage scenario is broken normative profiles that cannot be used without fixes. For example, some gematik profiles prior to package version 1.0.3-1 (note the suffix which does not obey FHIR/Semver syntax and semantics) were so broken that the HAPI validator crashed when trying to load them. Fixing these problems in the package cache (in order to be able to use some standard tools and libraries) requires that all tools leave already-cached packages alone. – DarthGizka Mar 14 '22 at 15:35
  • 1
    Appreciate the update and added context! We'll at least make it possible to turn off the auto-baking (the global setting I mentioned) to allow for these use cases. – Ward Weistra Mar 15 '22 at 08:54