2

There are a handful of things I'm not able to do because of my dependency on rustling-ontology:

  • cargo clippy needs my sources to be newer than my target or won't report anything - so it goes through my dependencies.
  • cargo tarpaulin wants to instrument everything through a full rebuild even though I don't want coverage information on my dependencies (and running it overwrites things in target/debug, so then cargo build wants to do a full rebuild too)
  • grcov doesn't work because there's some -sys crate that fails to build, so I can't get code coverage from it either.

Because of this build script and other dependencies (I think serde), cargo build takes nearly four minutes to complete.

I would ideally tell Cargo/Rust "here's a binary blob, just use that" instead of regretting every time I build.

I've tried to:

  • determine if I can package up the library binaries and ask Cargo to just use them instead of rebuilding.
  • Run cargo build -p my_package to ask Cargo to just leave whatever is built in target/ and use it.
  • export OUT_DIR= outside target/ - my hope was I could patch build.rs to just copy prebuilt training data in place. Cargo doesn't seem to recognize my OUT_DIR, and it looks like the package does some heavy lifting at build time with the output.
  • cargo vendor and try to snip some of the dependencies. Cargo notices I'm futzing with vendored code and is rightfully upset.
  • [patch."https://github.com/snipsco/rustling-ontology"] I tried to spin some of the dependencies in a local copy of this package, but other packages need the data I've snipped, and at some point I realized this isn't going to work.
  • tar cvf cached-target.tar target/ and untar the correct cache as I need it. This is the closest I've come to a working solution.
  • [UPDATED] Use sccache, but it seems like sccache is "smart" enough to require a rebuild since there's a build.rs associated.

I understand there are reasons why depending on precompiled libraries could break; but literally this is the same shell in the same directory of the same laptop, with the same system boot time as all the previous builds I've done; is there a release valve for this?

I'm using Rust 1.44.0-nightly 2020-04-09

cargo tree of the relevant dependencies (discussion of it follows):

my_example_app v0.1.0
├── snips-nlu-lib v0.65.6 (git+https://github.com/snipsco/snips-nlu-rs#88a047aacf40f3316a9cbdde850ea1798af39c2a)
    <some dependencies cut>
│   ├── base64 v0.10.1
│   ├── serde v1.0.106 (*)
│   ├── serde_json v1.0.51 (*)
│   ├── snips-nlu-ontology v0.67.1 (git+https://github.com/snipsco/snips-nlu-ontology?tag=0.67.1#194f99d87fa506ef1175b7dce70841c389940a29)
│   │   ├── failure v0.1.7 (*)
│   │   ├── serde v1.0.106 (*)
│   │   ├── serde_derive v1.0.106 (*)
│   │   └── serde_json v1.0.51 (*)
│   ├── snips-nlu-parsers v0.4.3 (git+https://github.com/snipsco/snips-nlu-parsers?tag=0.4.3#8cfdae50437732fcf906af90626c8db196729d30)
│   │   ├── failure v0.1.7 (*)
│   │   ├── gazetteer-entity-parser v0.8.0 (git+https://github.com/snipsco/gazetteer-entity-parser?tag=0.8.0#0e117818081782d1f295376f4cb3e1bb084e397c)
│   │   │   ├── failure v0.1.7 (*)
│   │   │   ├── fnv v1.0.6 (*)
│   │   │   ├── rmp-serde v0.13.7
│   │   │   │   ├── byteorder v1.3.4 (*)
│   │   │   │   ├── rmp v0.8.9
│   │   │   │   │   ├── byteorder v1.3.4 (*)
│   │   │   │   │   └── num-traits v0.2.11 (*)
│   │   │   │   └── serde v1.0.106 (*)
│   │   │   ├── serde v1.0.106 (*)
│   │   │   └── serde_json v1.0.51 (*)
│   │   ├── itertools v0.8.2 (*)
│   │   ├── lazy_static v1.4.0 (*)
│   │   ├── regex v1.3.6 (*)
│   │   ├── rustling-ontology v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0)
│   │   │   ├── rmp-serde v0.14.3
│   │   │   │   ├── byteorder v1.3.4 (*)
│   │   │   │   ├── rmp v0.8.9 (*)
│   │   │   │   └── serde v1.0.106 (*)
│   │   │   ├── rustling v0.9.1 (git+https://github.com/snipsco/rustling?tag=0.9.1#758074a27c339fa9e0cd477baee1f378fd0c2892)
│   │   │   │   ├── failure v0.1.7 (*)
│   │   │   │   ├── fnv v1.0.6 (*)
│   │   │   │   ├── rustling-core v0.9.1 (git+https://github.com/snipsco/rustling?tag=0.9.1#758074a27c339fa9e0cd477baee1f378fd0c2892)
│   │   │   │   │   ├── failure v0.1.7 (*)
│   │   │   │   │   ├── regex v1.3.6 (*)
│   │   │   │   │   ├── serde v1.0.106 (*)
│   │   │   │   │   ├── smallvec v0.6.13 (*)
│   │   │   │   │   └── string-interner v0.7.1
│   │   │   │   │       └── serde v1.0.106 (*)
│   │   │   │   ├── rustling-ml v0.9.1 (git+https://github.com/snipsco/rustling?tag=0.9.1#758074a27c339fa9e0cd477baee1f378fd0c2892)
│   │   │   │   │   ├── failure v0.1.7 (*)
│   │   │   │   │   ├── fnv v1.0.6 (*)
│   │   │   │   │   └── serde v1.0.106 (*)
│   │   │   │   └── serde v1.0.106 (*)
│   │   │   ├── rustling-ontology-grammar v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0)
│   │   │   │   ├── rustling v0.9.1 (git+https://github.com/snipsco/rustling?tag=0.9.1#758074a27c339fa9e0cd477baee1f378fd0c2892) (*)
│   │   │   │   ├── rustling-ontology-de v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0)
│   │   │   │   │   ├── rustling v0.9.1 (git+https://github.com/snipsco/rustling?tag=0.9.1#758074a27c339fa9e0cd477baee1f378fd0c2892) (*)
│   │   │   │   │   ├── rustling-ontology-moment v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0)
│   │   │   │   │   │   ├── chrono v0.4.8 (*)
│   │   │   │   │   │   ├── derive-new v0.5.8
│   │   │   │   │   │   │   ├── proc-macro2 v1.0.10 (*)
│   │   │   │   │   │   │   ├── quote v1.0.3 (*)
│   │   │   │   │   │   │   └── syn v1.0.17 (*)
│   │   │   │   │   │   ├── enum_primitive v0.1.1
│   │   │   │   │   │   │   └── num-traits v0.1.43
│   │   │   │   │   │   │       └── num-traits v0.2.11 (*)
│   │   │   │   │   │   ├── failure v0.1.7 (*)
│   │   │   │   │   │   └── vec_map v0.8.1 (*)
│   │   │   │   │   └── rustling-ontology-values v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0)
│   │   │   │   │       ├── failure v0.1.7 (*)
│   │   │   │   │       ├── log v0.4.8 (*)
│   │   │   │   │       ├── regex v1.3.6 (*)
│   │   │   │   │       ├── rustling v0.9.1 (git+https://github.com/snipsco/rustling?tag=0.9.1#758074a27c339fa9e0cd477baee1f378fd0c2892) (*)
│   │   │   │   │       └── rustling-ontology-moment v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   │   ├── rustling-ontology-en v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0)
│   │   │   │   │   ├── rustling v0.9.1 (git+https://github.com/snipsco/rustling?tag=0.9.1#758074a27c339fa9e0cd477baee1f378fd0c2892) (*)
│   │   │   │   │   ├── rustling-ontology-moment v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   │   │   └── rustling-ontology-values v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   │   ├── rustling-ontology-es v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0)
│   │   │   │   │   ├── rustling v0.9.1 (git+https://github.com/snipsco/rustling?tag=0.9.1#758074a27c339fa9e0cd477baee1f378fd0c2892) (*)
│   │   │   │   │   ├── rustling-ontology-moment v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   │   │   └── rustling-ontology-values v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   │   ├── rustling-ontology-fr v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0)
│   │   │   │   │   ├── rustling v0.9.1 (git+https://github.com/snipsco/rustling?tag=0.9.1#758074a27c339fa9e0cd477baee1f378fd0c2892) (*)
│   │   │   │   │   ├── rustling-ontology-moment v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   │   │   └── rustling-ontology-values v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   │   ├── rustling-ontology-it v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0)
│   │   │   │   │   ├── rustling v0.9.1 (git+https://github.com/snipsco/rustling?tag=0.9.1#758074a27c339fa9e0cd477baee1f378fd0c2892) (*)
│   │   │   │   │   ├── rustling-ontology-moment v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   │   │   └── rustling-ontology-values v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   │   ├── rustling-ontology-ja v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0)
│   │   │   │   │   ├── regex v1.3.6 (*)
│   │   │   │   │   ├── rustling v0.9.1 (git+https://github.com/snipsco/rustling?tag=0.9.1#758074a27c339fa9e0cd477baee1f378fd0c2892) (*)
│   │   │   │   │   ├── rustling-ontology-moment v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   │   │   └── rustling-ontology-values v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   │   ├── rustling-ontology-ko v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0)
│   │   │   │   │   ├── failure v0.1.7 (*)
│   │   │   │   │   ├── regex v1.3.6 (*)
│   │   │   │   │   ├── rustling v0.9.1 (git+https://github.com/snipsco/rustling?tag=0.9.1#758074a27c339fa9e0cd477baee1f378fd0c2892) (*)
│   │   │   │   │   ├── rustling-ontology-moment v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   │   │   └── rustling-ontology-values v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   │   ├── rustling-ontology-pt v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0)
│   │   │   │   │   ├── rustling v0.9.1 (git+https://github.com/snipsco/rustling?tag=0.9.1#758074a27c339fa9e0cd477baee1f378fd0c2892) (*)
│   │   │   │   │   ├── rustling-ontology-moment v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   │   │   └── rustling-ontology-values v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   │   ├── rustling-ontology-values v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   │   └── rustling-ontology-zh v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0)
│   │   │   │       ├── rustling v0.9.1 (git+https://github.com/snipsco/rustling?tag=0.9.1#758074a27c339fa9e0cd477baee1f378fd0c2892) (*)
│   │   │   │       ├── rustling-ontology-moment v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   │       └── rustling-ontology-values v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   ├── rustling-ontology-moment v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   ├── rustling-ontology-values v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   └── serde v1.0.106 (*)
│   │   │   [build-dependencies]
│   │   │   ├── rmp-serde v0.14.3 (*)
│   │   │   ├── rustling v0.9.1 (git+https://github.com/snipsco/rustling?tag=0.9.1#758074a27c339fa9e0cd477baee1f378fd0c2892) (*)
│   │   │   ├── rustling-ontology-grammar v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   ├── rustling-ontology-moment v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   ├── rustling-ontology-values v0.19.3 (git+https://github.com/snipsco/rustling-ontology?tag=0.19.3#3bb1313de543aa2a8027da9fc11c435db0cce8c0) (*)
│   │   │   └── serde v1.0.106 (*)
<more dependencies cut>

rustling-ontology brings in rustling-ontology-grammar, then a number of language specific crates. The crate has a build.rs that trains the language model for each of the available languages. I believe it then takes the output of that and uses serde to generate actual Rust code which is compiled at build time. This process alone takes a few minutes and has a number of *-sys dependencies.

Issues:

  1. grcov does not work with some of the sys dependencies within rustling-ontology; it gives me errors about compiling C code. I'm sure I could go spelunking in GitHub, but the truth is - I don't want grcov to cover anything related to rustling-ontology (or any of those subtree of packages).

  2. cargo tarpaulin / cargo build / cargo build --release / cargo clippy. Clippy requires a full rebuild (due to bugs) to provide useful information. Since I'm not going to fix any lint issues in there, I don't want Clippy to go through rustling-ontology and have to wait for minutes while it retrains and re-serdes. Tarpaulin also does a full rebuild, and it seems like it crosses wires with the debug build on occasion, so just getting code coverage on unit tests takes minutes. Using -p to scope to a specific package doesn't help.

Reproduction

You'll notice that rustling-ontology takes quite a while to build if you run:

$ git clone https://github.com/snipsco/snips-nlu-rs
$ cd snips-nlu-rs
$ cargo run --example interactive_parsing_cli data/tests/models/nlu_engine

I guess in a nutshell, I would like to have some library dependencies brought in only as pre-compiled binaries into my executable.

Idcmp
  • 659
  • 1
  • 8
  • 19
  • Your problem might be solved by [Out of source builds (external build directory) with Cargo?](https://stackoverflow.com/q/41274901/155423). – Shepmaster Apr 14 '20 at 13:46
  • It's hard to answer your question because it doesn't include a [MRE]. We can't tell what crates (and their versions), types, traits, fields, etc. are present in the code. It would make it easier for us to help you if you try to reproduce your error on the [Rust Playground](https://play.rust-lang.org) if possible, otherwise in a brand new Cargo project, then [edit] your question to include the additional info. There are [Rust-specific MRE tips](//stackoverflow.com/tags/rust/info) you can use to reduce your original code for posting here. Thanks! – Shepmaster Apr 14 '20 at 13:46
  • It's not at all clear what the problem exactly is. The build takes a while, but builds are cached. Presumably you are doing *something* that blows the cache, but since there's no exact steps to reproduce, we can't see exactly what it is or how it's failing the cache, so we can't provide useful feedback. – Shepmaster Apr 14 '20 at 13:48
  • Sorry, my question was edited, so some key things aren't as visible. I'll do a sweep. – Idcmp Apr 14 '20 at 17:50
  • FWIW, [*I* performed the edits](https://stackoverflow.com/posts/61199511/revisions) in an attempt to understand your meaning. Since that failed, I asked for the steps to reproduce the problem, as code will ideally show what the prose does not. – Shepmaster Apr 14 '20 at 17:51
  • More details, and an example "slow build" added. – Idcmp Apr 14 '20 at 18:18
  • Builds are cached — `% time cargo build -> 4:26.99` then `% time cargo build -> 0.316`. So yes, you have to wait 4 minutes once per build type (`check` vs `build` or `--release` vs not), but only once. – Shepmaster Apr 14 '20 at 19:07
  • *Clippy requires a full rebuild (due to bugs)* — does using [the suggestion in my first comment](https://stackoverflow.com/questions/61199511/how-do-i-reduce-or-speed-up-rebuilds-when-using-rustling-ontology#comment108283262_61199511) help? – Shepmaster Apr 14 '20 at 19:09
  • It took me a bit to understand how this would all work together, but I think I understand it now. I'll give this a try tonight. – Idcmp Apr 14 '20 at 22:05
  • To be exact, you're suggesting I make a new crate, that has rustling-ontology (would actually be "snips-nlu-lib" as "rustling-ontology" comes in transitively) as a dependency, but shares the same target directory. Then, instead of depending on "snips-nlu-lib", depend on my new crate in my project, and this should prevent rustling-ontology from being rebuilt? I'm confused because right now that crate comes in as a dependency anyway? – Idcmp Apr 15 '20 at 03:21

1 Answers1

1

At this time, there's no way to speed things up that work with the way rustling-ontology is built.

If you've stumbled across this looking for ways to "make your development cycle shorter", then my recommendation is to looks at sccache which had the best impact, but specifically wouldn't cache rustling-ontology.

Idcmp
  • 659
  • 1
  • 8
  • 19