Ergotree proc-macro

[kettlebell]

Close #633

[kettlebell]

@greenhat, after a rebase I get the follow GA build error for ubuntu: https://github.com/ergoplatform/sigma-rust/actions/runs/3180706098/jobs/5185687305#step:5:745

Have you seen this before? Not sure why it's happening. It just appeared after rebasing to develop.

[greenhat]

@greenhat, after a rebase I get the follow GA build error for ubuntu: https://github.com/ergoplatform/sigma-rust/actions/runs/3180706098/jobs/5185687305#step:5:745

Have you seen this before? Not sure why it's happening. It just appeared after rebasing to develop.

I've never seen anything like this. Sounds like a rustc bug to me.

[kettlebell]

I've never seen anything like this. Sounds like a rustc bug to me.

It's happening on my mac too. It's very strange but it looks like it manifests only when testing in release mode (plain cargo test works fine). I think it's related to these warnings at the beginning https://github.com/ergoplatform/sigma-rust/actions/runs/3180706098/jobs/5185687305#step:5:11. I looked up the associated rust issue, but can't really make sense of it.

There was a comment in the issue suggesting to first run cargo build --release first. But this gives the following cryptic error:

   Compiling ergotree-macro v0.20.0 (/Users/tling/work/rust_projects/sigma-rust/ergotree-macro)
error[E0460]: found possibly newer version of crate `ergo_chain_types` which `ergotree_ir` depends on
 --> ergotree-macro/src/lib.rs:4:5
  |
4 | use ergotree_ir::mir::expr::Expr;
  |     ^^^^^^^^^^^
  |
  = note: perhaps that crate needs to be recompiled?
  = note: the following crate versions were found:
          crate `ergo_chain_types`: .../sigma-rust/target/release/deps/libergo_chain_types.rlib
          crate `ergotree_ir`: .../sigma-rust/target/release/deps/libergotree_ir.rlib

[greenhat]

It does not ring any bells. I, too, think the warnings on duplicate artifacts could be connected.

[kettlebell]

I've tried my best given limited time but I haven't been able to fully understand the problem. However I have a workaround that lets us proceed. The main problem is that we cannot run build/test with --release flag from the root directory of this repo. Leads to those cryptic errors.

In this PR I've created a sub-crate called ergotree-macro to hold the proc-macro. If you take a look it's super simple, but it needs to refer to ergotree_ir::mir::expr::Expr, so it has a dependency on ergotree-ir. Now all the hard work to generate the Expr instances is done within ergotree-ir, where various types implement traits that are necessary for the proc-macro. I've gated all these implementations behind a feature flag ergotree-proc-macro, so it will only be pulled in when we run our JIT test suite. This feature flag is activated for ergotree-macro, which I think is the source of all the trouble, because all the other crates that depend on ergotree-ir don't need this feature.

Anyways, the workaround is to run each test suite separately, and not as a monolithic cargo build --release --tests from the root. So to run the test suite for ergotree-ir in release mode we execute from the root directory:

cargo test --release --all-features --manifest-path ergotree-ir/Cargo.toml

And we do a similar thing for the other crates. Now another strange thing that happens is the test suite for ergotree-macro can't even build in release mode BUT it will run in debug. Have no idea why. But I think it's fine to run these tests in debug mode.

[greenhat]

Good job! The separate crate for macros does ring a bell, but I'm struggling to recall the exact reason.
I don't think --release is that important for tests. If running them in debug build does not require calling each crate, I'd rather switch to debug.

[kettlebell]

Good job! The separate crate for macros does ring a bell, but I'm struggling to recall the exact reason. I don't think --release is that important for tests. If running them in debug build does not require calling each crate, I'd rather switch to debug.

Ok great, I'll just remove --release for the tests. Debug only sees major slowdowns for tests in gf2_192, from what I recall.

The rust compiler has special requirements for proc-macro. The crate that contains it must have

[lib]
proc-macro = true

set within its Cargo.toml. I'm not exactly sure myself why, but this answer gives some possible reasons: https://stackoverflow.com/a/56714256

[greenhat]

The rust compiler has special requirements for proc-macro. The crate that contains it must have

[lib]
proc-macro = true

set within its Cargo.toml. I'm not exactly sure myself why, but this answer gives some possible reasons: https://stackoverflow.com/a/56714256

Thank you! After reading your link I recalled that from my Scala macros experience. The procedural macros should be compiled before the compilation unit where it's called. The compiler plugin is a good mental model for this.

[kettlebell]

Thank you! After reading your link I recalled that from my Scala macros experience. The procedural macros should be compiled before the compilation unit where it's called. The compiler plugin is a good mental model for this.

Yeah I agree. Does ergo make use of scala macros?

[greenhat]

Thank you! After reading your link I recalled that from my Scala macros experience. The procedural macros should be compiled before the compilation unit where it's called. The compiler plugin is a good mental model for this.

Yeah I agree. Does ergo make use of scala macros?

I don't think node uses macros at all. I wrote a lot of macros in https://github.com/ergoplatform/ergo-scala-compiler . Check out the readme with an example of contract compilation. Since ErgoScript is mostly a subset of Scala, we can write a contract in Scala and compile it directly into an ErgoTree.

[kettlebell]

I don't think node uses macros at all. I wrote a lot of macros in https://github.com/ergoplatform/ergo-scala-compiler . Check out the readme with an example of contract compilation. Since ErgoScript is mostly a subset of Scala, we can write a contract in Scala and compile it directly into an ErgoTree.

Thanks! Looks really cool. And also verification using SMT solvers! 🤯

[greenhat]

Those were the fun times. :) The ErgoScala code verification is an earlier project of mine - https://github.com/ergoplatform/ergo-contracts . I just used Stainless https://stainless.epfl.ch/ to prove a Scala code equivalent(ErgoScala) of the ErgoScript contract.

[kettlebell]

Those were the fun times. :) The ErgoScala code verification is an earlier project of mine - https://github.com/ergoplatform/ergo-contracts . I just used Stainless https://stainless.epfl.ch/ to prove a Scala code equivalent(ErgoScala) of the ErgoScript contract.

Awesome, do you have plans to return to the project(s)?

[kettlebell]

I'm leaving this PR in its current form as it serves as my work-submission for Ergohack. I'll continue things by making a PR on top of this.

[greenhat]

Those were the fun times. :) The ErgoScala code verification is an earlier project of mine - https://github.com/ergoplatform/ergo-contracts . I just used Stainless https://stainless.epfl.ch/ to prove a Scala code equivalent(ErgoScala) of the ErgoScript contract.

Awesome, do you have plans to return to the project(s)?

Not in the near future. I'd be happy to carve up some time for ErgoScala and use it to power contract development with debugging and Scala static analysis tools. Code verification needs way too much time.