Standard Circom Gadget with Keccak example #997

tchataigner · 2023-12-21T17:29:09Z

Goal of this PR

The goal of this PR is to properly implement some standard Circom Gadget in our Lurk library, along with a Keccak example.

Current progress

Implemented two ways for the CircomCoprocessor to look for gadget, either via CLI or directly on a remote Github repository
Create a standard template for Circom Gadget on Github
- Help required to pass it in public instead of private
Create keccak-circom-gadget repository based on the template.

Left TODO

Create pull request over circom-scotia to ensure that we can have slice returned as an output. Branch here.
Once merged, update the example for Keccak in lurk-rs to make it work
Finally, derive the example to implement the CircomKeccak implementing CircomGadget directly in our repository.

Current caveats

The goal of the following bullet points is to point out diverse part of the flow where I think I need external opinion before setting them in stone :)

Right now, the CircomCoprocessor looks for the CircomGadget files locally before fetching them from the remote rpeository. This is a great simple caching solution BUT it might be too naïve because it does not emcompasss the version of the gadget. Thus if I update my CircomGadget version it will still use the old one. Should I fix that?
I'd like to know if what I've done for the foundation of remote Circom Gadget seems alright.

Related issue

This is part of tackling the issue #962

huitseeker

I like the approach of grabbing the files from an online location: even if I'm a bit sceptical about the bit about doing, specifically, a Github release, which I feel might add friction.

Have you created a small toy repo where we'd be able to test this import mechanism? Perhaps try to run a GH import on nightly?

src/cli/circom.rs

huitseeker · 2023-12-22T12:53:50Z

src/cli/circom.rs

+pub(crate) fn create_circom_gadget(circom_folder: &Utf8PathBuf, reference: &str) -> Result<()> {
+    let circom_gadget = circom_dir().join(reference);
+
+    // We expect a format <AUTHOR>/<NAME> for the name.


nit: GADGET_NAME may be more useful, otherwise the reader might just wonder who this is supposed to be the name of. (nit -> you can safely ignore this)

src/cli/mod.rs

src/coprocessor/circom/README.md

src/coprocessor/circom/mod.rs

huitseeker · 2023-12-22T18:13:22Z

src/coprocessor/circom/mod.rs

+        /// TODO: Generalize
+        fn arity(&self) -> usize {
+            0
+        }
+
+        fn synthesize_simple<CS: ConstraintSystem<F>>(
+            &self,
+            cs: &mut CS,
+            g: &crate::lem::circuit::GlobalAllocator<F>,
+            _s: &Store<F>,
+            _not_dummy: &bellpepper_core::boolean::Boolean,
+            args: &[AllocatedPtr<F>],
+        ) -> std::result::Result<AllocatedPtr<F>, SynthesisError> {
+            let input = self.gadget.clone().into_circom_input(args);
+            let witness =
+                circom_scotia::calculate_witness(&self.config, input, true).map_err(|e| {
+                    eprintln!("{:?}", e);
+                    SynthesisError::Unsatisfiable
+                })?;
+            let output = circom_scotia::synthesize(cs, self.config.r1cs.clone(), Some(witness))?;
+            let num_tag = g.alloc_tag(cs, &crate::tag::ExprTag::Num);
+            let res = AllocatedPtr::from_parts(num_tag.clone(), output);
+
+            Ok(res)
+        }
+    }
+
+    impl<F: LurkField, C: CircomGadget<F> + Debug> Coprocessor<F> for CircomCoprocessor<F, C> {
+        /// TODO: Generalize
+        fn eval_arity(&self) -> usize {
+            0
+        }


Yep, those todos are going to be the next steps to improve on.

tchataigner · 2023-12-22T18:30:52Z

Have you created a small toy repo where we'd be able to test this import mechanism? Perhaps try to run a GH import on nightly?

Not 100% sure about what you mean but I have adapted the template in a keccak example. The keccak example is also using this remote logic.

I like the approach of grabbing the files from an online location: even if I'm a bit sceptical about the bit about doing, specifically, a Github release, which I feel might add friction.

Would you have a recommendation of another place to host them?

huitseeker · 2023-12-22T18:50:06Z

I have adapted the template in a keccak example. The keccak example is also using this remote logic.

Oh, nice!

Would you have a recommendation of another place to host them?

For us at Lurk Lab? I'm fine with releases, for now (those gadgets can get huge)

But in I was thinking user may just give us 2 URLs to their S3 bucket, and perhaps we could work with that. OTOH your template repo completely removes any issue re: friction, only the price of hosting large files on GH remains as an issue.

arthurpaulino

This is amazing work! I can see that it's already quite polished.

Suggestion: print the output of the computation in the Keccak example so we can see the hash that Lurk spits out. The output is in the last frame, in the first component of the output attribute. You can print it with the fmt_to_string_simple method.

examples/keccak.rs

src/lem/circuit.rs

arthurpaulino

🎉

huitseeker

This is excellent, I <3 the careful error messages. I left a few inline comments, nothing blocking. Thank you so much!

huitseeker · 2024-03-11T21:43:34Z

examples/keccak.rs

+fn bytes_to_bits(bytes: &[u8]) -> Vec<bool> {
+    let mut bits = Vec::new(); // Create a new, empty vector to store bits
+
+    for &byte in bytes.iter() {
+        // Iterate over each byte in the input slice
+        for j in 0..8 {
+            // For each bit in the byte
+            if byte & (1 << j) > 0 {
+                // Check if the bit is set
+                bits.push(true); // If the bit is set, push 1 to the vector
+            } else {
+                bits.push(false); // If the bit is not set, push 0
+            }
+        }
+    }
+    bits // Return the vector of bits
+}


Could we just import https://github.com/lurk-lab/bellpepper/blob/02cdf17607a4a31ea894e6b0fe9b21e6e6bc9331/crates/bellpepper/src/gadgets/multipack.rs#L40-L45?

Switched to bytes_to_bit_le

huitseeker · 2024-03-11T21:44:51Z

examples/keccak.rs

+fn bits_to_bytes(bits: &[bool]) -> Vec<u8> {
+    let mut bytes = vec![0; (bits.len() + 7) / 8]; // Initialize a vector with zeroes
+
+    for (i, &bit) in bits.iter().enumerate() {
+        // Iterate over each bit with its index
+        let byte_index = i / 8; // Calculate the byte index for the current bit
+        if bit {
+            // If the current bit is true,
+            bytes[byte_index] |= 1 << (i % 8); // Set the corresponding bit in the byte
+        }
+    }
+    bytes // Return the array of bytes
+}


Could we just import https://github.com/lurk-lab/neptune/blob/2e9ee85c888d2e7ca2bd225060f23586cb7f518d/src/round_constants.rs#L181-L194 ?

This seems to be only available in the crate context, not publicly

src/coprocessor/circom/mod.rs

huitseeker

Thanks a ton!

- With the new implementation, we first try to fetc hthe gadget locally and fallback on remote when needed. - Introduced a version over Circom Gadgets. Useful to point to a particular remote release. - Better error propagation - Precise error message based on where the flow is failing

This is WIP as there is a need to fix circom-scotia before being able to make the Keccak example work. - Update framework -> stack in the README - Create an example skeleton for a remote Keccak example - Fix the circom example to use an imported Circom Gadget from the CLI

- Removed deleted keccak mod - Fixed clippy warning

- Adapted circom-scotia to use main instead of dev - Created a structure, CircomReference to qualify an identifier for CircomGadget - Better check on CircomReference initialization

Arity is now inherited directly from the gadget, making it declarative by the gadget developer.

Co-authored-by: François Garillot <4142+huitseeker@users.noreply.github.com>

tchataigner · 2024-03-13T08:27:33Z

@arthurpaulino thanks for noticing! Fixed this!

tchataigner requested review from huitseeker and winston-h-zhang December 21, 2023 19:32

tchataigner force-pushed the feature/keccac-coproc branch from e84369e to 224004d Compare December 21, 2023 20:51

huitseeker reviewed Dec 22, 2023

View reviewed changes

tchataigner mentioned this pull request Jan 18, 2024

Make Circom coprocessor use arguments #789

Open

huitseeker mentioned this pull request Feb 20, 2024

chore: make circom-scotia depend on released version of bellpepper-core argumentcomputer/circom-scotia#30

Merged

tchataigner force-pushed the feature/keccac-coproc branch from cc59ac1 to fe798a8 Compare February 23, 2024 16:25

tchataigner force-pushed the feature/keccac-coproc branch 2 times, most recently from 4620c24 to 8642a16 Compare March 7, 2024 11:35

tchataigner marked this pull request as ready for review March 8, 2024 15:57

tchataigner requested review from a team as code owners March 8, 2024 15:57

arthurpaulino reviewed Mar 8, 2024

View reviewed changes

examples/keccak.rs Outdated Show resolved Hide resolved

examples/keccak.rs Show resolved Hide resolved

examples/keccak.rs Show resolved Hide resolved

tchataigner force-pushed the feature/keccac-coproc branch from 518b243 to 4e7b8ce Compare March 8, 2024 19:01

tchataigner requested review from arthurpaulino and Nramsrud March 8, 2024 19:16

arthurpaulino reviewed Mar 8, 2024

View reviewed changes

src/lem/circuit.rs Outdated Show resolved Hide resolved

tchataigner requested a review from arthurpaulino March 8, 2024 20:50

arthurpaulino previously approved these changes Mar 8, 2024

View reviewed changes

tchataigner requested review from huitseeker, gabriel-barrett and adr1anh March 8, 2024 21:10

tchataigner dismissed arthurpaulino’s stale review via aea2b62 March 11, 2024 15:56

huitseeker previously approved these changes Mar 11, 2024

View reviewed changes

tchataigner dismissed huitseeker’s stale review via 551f052 March 12, 2024 16:35

tchataigner requested a review from huitseeker March 12, 2024 17:21

huitseeker previously approved these changes Mar 12, 2024

View reviewed changes

tchataigner and others added 20 commits March 13, 2024 09:14

docs(circom): README to explain behavior of Circom Coproc & Gadget

3e7c781

docs(circom): added link to template gadget

0630e29

fix(circom): fixed CI test

6bb752c

- Removed deleted keccak mod - Fixed clippy warning

fix(circom): fixed clippy in examples

9ac4a17

feat(circom): wip review

db3e647

- Adapted circom-scotia to use main instead of dev - Created a structure, CircomReference to qualify an identifier for CircomGadget - Better check on CircomReference initialization

feat(circom): wip argument from lurk

e6efcff

Arity is now inherited directly from the gadget, making it declarative by the gadget developer.

feat(circom): trying out AllocatedPtr and Var

1abf895

feat(circom): wip working on evaluation values

e86ee6e

refactor(circom): wip making list work

946d57e

feature(circom): working keccak example

c071a6c

feature(circom): only keeping one example

7ba3139

refactor(circom): fix xclippy

6067c4c

refactor(circom): fix wasm build

a45b636

refactor(circom): integrate review & adapt chain_car_cdr

c941350

refactor(circom): remove unecessary todos

ccaec9b

refactor(circom): to bits le on alloc num

cca3bf7

Update src/coprocessor/circom/mod.rs

0d77ddd

Co-authored-by: François Garillot <4142+huitseeker@users.noreply.github.com>

refactor(circom): using bellpepper utils

85d4897

tchataigner force-pushed the feature/keccac-coproc branch from c5a5103 to 85d4897 Compare March 13, 2024 08:17

fix(circom): using proper commit for lurk lib

faaf71d

tchataigner dismissed huitseeker’s stale review via faaf71d March 13, 2024 08:27

tchataigner requested a review from huitseeker March 13, 2024 08:27

huitseeker approved these changes Mar 13, 2024

View reviewed changes

tchataigner added this pull request to the merge queue Mar 13, 2024

Merged via the queue into argumentcomputer:main with commit ddf26a1 Mar 13, 2024
11 checks passed

tchataigner deleted the feature/keccac-coproc branch March 13, 2024 23:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Standard Circom Gadget with Keccak example #997

Standard Circom Gadget with Keccak example #997

tchataigner commented Dec 21, 2023 •

edited

Loading

huitseeker left a comment

huitseeker Dec 22, 2023

huitseeker Dec 22, 2023

tchataigner commented Dec 22, 2023

huitseeker commented Dec 22, 2023 •

edited

Loading

arthurpaulino left a comment

arthurpaulino left a comment

huitseeker left a comment

huitseeker Mar 11, 2024

tchataigner Mar 12, 2024

huitseeker Mar 11, 2024

tchataigner Mar 12, 2024

huitseeker left a comment

tchataigner commented Mar 13, 2024

Standard Circom Gadget with Keccak example #997

Standard Circom Gadget with Keccak example #997

Conversation

tchataigner commented Dec 21, 2023 • edited Loading

Goal of this PR

Current progress

Left TODO

Current caveats

Related issue

huitseeker left a comment

Choose a reason for hiding this comment

huitseeker Dec 22, 2023

Choose a reason for hiding this comment

huitseeker Dec 22, 2023

Choose a reason for hiding this comment

tchataigner commented Dec 22, 2023

huitseeker commented Dec 22, 2023 • edited Loading

arthurpaulino left a comment

Choose a reason for hiding this comment

arthurpaulino left a comment

Choose a reason for hiding this comment

huitseeker left a comment

Choose a reason for hiding this comment

huitseeker Mar 11, 2024

Choose a reason for hiding this comment

tchataigner Mar 12, 2024

Choose a reason for hiding this comment

huitseeker Mar 11, 2024

Choose a reason for hiding this comment

tchataigner Mar 12, 2024

Choose a reason for hiding this comment

huitseeker left a comment

Choose a reason for hiding this comment

tchataigner commented Mar 13, 2024

tchataigner commented Dec 21, 2023 •

edited

Loading

huitseeker commented Dec 22, 2023 •

edited

Loading