Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement persistent commitments #543

Merged
merged 20 commits into from
Jul 20, 2023
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion fcomm/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -555,7 +555,7 @@ impl<F: LurkField + Serialize + DeserializeOwned> LurkPtr<F> {
LurkPtr::ZStorePtr(z_store_ptr) => {
let z_store = &z_store_ptr.z_store;
let z_ptr = z_store_ptr.z_ptr;
s.intern_z_expr_ptr(z_ptr, z_store)
s.intern_z_expr_ptr(&z_ptr, z_store)
.expect("failed to intern z_ptr")
}
}
Expand Down
44 changes: 44 additions & 0 deletions src/cli/commitment.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
use lurk::{field::LurkField, z_ptr::ZExprPtr, z_store::ZStore};
use serde::{Deserialize, Serialize};

/// Holds data for commitments.
///
/// **Warning**: holds private data. The `ZStore` contains the secret used to
/// hide the original payload.
#[derive(Serialize, Deserialize)]
pub struct Commitment<F: LurkField> {
pub(crate) hidden: ZExprPtr<F>,
pub(crate) zstore: ZStore<F>,
}

#[cfg(not(target_arch = "wasm32"))]
mod cli {
use anyhow::Result;
use lurk::{field::LurkField, ptr::Ptr, store::Store, z_store::ZStore};
use serde::Serialize;
use std::{fs::File, io::BufWriter};

use crate::cli::{field_data::FieldData, paths::cli::commitment_path};

use super::Commitment;

impl<F: LurkField> Commitment<F> {
pub fn new(secret: F, payload: Ptr<F>, store: &mut Store<F>) -> Result<Self> {
let hidden = store.hide(secret, payload);
let mut zstore = Some(ZStore::<F>::default());
let hidden = store.get_z_expr(&hidden, &mut zstore)?.0;
Ok(Self {
hidden,
zstore: zstore.unwrap(),
})
}
}

impl<F: LurkField + Serialize> Commitment<F> {
pub fn persist(&self, hash: &str) -> Result<()> {
let fd = &FieldData::wrap::<F, Commitment<F>>(self)?;
bincode::serialize_into(BufWriter::new(&File::create(commitment_path(hash))?), fd)?;
Ok(())
}
}
}
30 changes: 30 additions & 0 deletions src/cli/field_data.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
use anyhow::{bail, Result};
use serde::{Deserialize, Serialize};

use lurk::field::{LanguageField, LurkField};

/// A wrapper for data whose deserialization depends on a certain LurkField
#[derive(Serialize, Deserialize)]
pub struct FieldData {
pub(crate) field: LanguageField,
data: Vec<u8>,
huitseeker marked this conversation as resolved.
Show resolved Hide resolved
}

#[allow(dead_code)]
impl FieldData {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, but with this version you are moving the conversion to/from bytes at the wrapping time. I suspect you could peg what you want on a

struct Labeled<T: Serialize + DeserializeOwned> {
  label: Language field,
  val: T,
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this case I do intend to deserialize in two steps. We first read from FS to know the field and then we read from the vector to get the data with the correct field elements

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. You can definitely have those two steps, but in sequence within the same function, with the above structure.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't get the idea. I don't want to deserialize the vector of bytes if there is some inconsistency in the field. I want to error earlier. In other words, the vector is desirable

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the record, I did try something like you propose in my first attempt but then I got stuck because Rust doesn't have dependent types. I wanted T<F> where F: LurkField

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm.

As partially unrelated topic, I don't think any strategy that persists important Lurk data using an ad-hoc/bincode serialization of LanguageField is a good idea. It would be A Bad Thing if changes to that enum (which are so likely they are predictable, so predictable we should plan for them) led to supposedly durable data becoming unreadable.

Copy link
Member Author

@arthurpaulino arthurpaulino Jul 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can we encode that information, then? Should we create another enum that we try to assure its stability?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you want to have a completely general, 'dynamic' serialization, then it's going to require more design.

But is that really what is needed here? I think you just need to know what LanguageField you are currently using. Then everything will be written and read using that LanguageField. Moreover, it follows from our general cryptographic assumptions that any value used as a commitment (or as the hash part of a Lurk expression) cannot be produced by hashing some preimage in another LanguageField.

Therefore, as long as you are looking values up by their hash/digest, then it's fine to completely segregate them by field. So, given that this work is still using a filesystem-based commitment store, you could have a directory structure like:

.lurk/commitments/pallas
.lurk/commitments/vesta
.lurk/commitments/bls12-81
.lurk/commitments/grumpkin
…

If the current LanguageField is pallas, then you can write all commitments to the .lurk/commitments/pallas directory and look all commitments up there too. There is never any possibility that a valid commitment (say you are looking it up) expressed as an element of pallas::Scalar will be stored elsewhere.

Also, since (as above), we also cannot have collisions between fields (assuming indexes are always hashes, and the chosen field/hash combinations still preserve our security assumptions) you could even get the 'dynamic' behavior by searching for a given commitment in all available field directories. To prevent having to search in multiple places, you could use symlinks (for example) to provide a single index (perhaps hierarchically structured to avoid too-large directories, etc.)

Obviously, with a more powerful database management system than 'the file system', a different approach could be taken. But I think the above (especially the simplest version, which is probably all we need initially) should be fine.

The point is: I think you may be trying to solve the wrong problem. Certainly if the goal is a quick PR that decrees a format through code then that is the case.

While we may want to eventually have a format that allows dynamically mixing field types, that will need to be worked out as a careful extension of z_data.

Copy link
Member Author

@arthurpaulino arthurpaulino Jul 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That works for commitments, but lurk verify <proof_id> would need to ask the user for the field, which is annoying. So I thought that we might as well just use the same infra that's already available to give us extra consistency, assuring that we won't open a commitment that was generated in a field while we're in another field.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aren't proofs also always with respect to a known prover/verifier, which is itself necessarily parameterized on curve+field?

Also, why the heck are proofs (apparently) being stored with an id that is the timestamp rather than using content-addressing as previously?


Question: Are you trying to support verifying any Lurk proof, or do we assume that a given REPL session will only verify proofs of statements in the current field?

Either way, if you content-address the proofs, you should be able to use the symlink approach as above. (NOTE: in that model, you will need to check the actual location of resolved symlinks to get that meta-data — but I still think that's not what should be needed here.)


Opinion: Proof IDs should be the content address of the claim — just as they are in fcomm and clutch. This is actually important because it allows for caching of proofs. It's easy to imagine applications for which equivalent proofs are requested more than once (even many, many times). For example, that's how the current Lurk website works (or was designed to): we serve real proofs of expected claims in a way that is cost effective but still accurate.

A real outsourced-but-provable computation service could do the same.

#[inline]
pub fn wrap<F: LurkField, T: Serialize>(t: &T) -> Result<Self> {
Ok(Self {
field: F::FIELD,
data: bincode::serialize(t)?,
})
}

#[inline]
pub fn extract<'a, F: LurkField, T: Deserialize<'a>>(&'a self) -> Result<T> {
if self.field != F::FIELD {
bail!("Invalid field: {}. Expected {}", &self.field, &F::FIELD)
}
Ok(bincode::deserialize(&self.data)?)
}
}
152 changes: 82 additions & 70 deletions src/cli/lurk_proof.rs
Original file line number Diff line number Diff line change
@@ -1,61 +1,43 @@
use serde::{Deserialize, Serialize};

use anyhow::Result;

use lurk::{
coprocessor::Coprocessor,
eval::{
lang::{Coproc, Lang},
Status,
},
field::{LanguageField, LurkField},
field::LurkField,
proof::nova,
public_parameters::public_params,
z_ptr::ZExprPtr,
z_store::ZStore,
};

/// A wrapper for data whose deserialization depends on a certain LurkField
#[derive(Serialize, Deserialize)]
pub struct FieldData {
field: LanguageField,
data: Vec<u8>,
}

#[allow(dead_code)]
impl FieldData {
#[inline]
pub fn wrap<F: LurkField, T: Serialize>(t: &T) -> Result<Self> {
Ok(Self {
field: F::FIELD,
data: bincode::serialize(t)?,
})
}

#[inline]
pub fn open<'a, T: Deserialize<'a>>(&'a self) -> Result<T> {
Ok(bincode::deserialize(&self.data)?)
}
}

/// Carries extra information to help with visualization, experiments etc
/// Carries extra information to help with visualization, experiments etc.
///
/// Note: the `ZStore` in this struct only has enough data to recover the meaning
/// of the claim being proven: `expression`, when evaluated in the context of
/// `environment`, is reduced to `result`. It doesn't contain private data.
#[derive(Serialize, Deserialize)]
pub struct LurkProofMeta<F: LurkField> {
pub iterations: usize,
pub evaluation_cost: u128,
pub generation_cost: u128,
pub compression_cost: u128,
pub status: Status,
pub expression: ZExprPtr<F>,
pub environment: ZExprPtr<F>,
pub result: ZExprPtr<F>,
pub zstore: ZStore<F>,
pub(crate) iterations: usize,
pub(crate) evaluation_cost: u128,
pub(crate) generation_cost: u128,
pub(crate) compression_cost: u128,
pub(crate) status: Status,
pub(crate) expression: ZExprPtr<F>,
pub(crate) environment: ZExprPtr<F>,
pub(crate) result: ZExprPtr<F>,
pub(crate) zstore: ZStore<F>,
}

type F = pasta_curves::pallas::Scalar; // TODO: generalize this
type Pallas = pasta_curves::pallas::Scalar; // TODO: generalize this

/// Minimal data structure containing just enough for proof verification
#[derive(Serialize, Deserialize)]
pub enum LurkProof<'a> {
pub enum LurkProof<'a, F: LurkField>
where
Coproc<F>: Coprocessor<Pallas>,
{
Nova {
proof: nova::Proof<'a, Coproc<F>>,
public_inputs: Vec<F>,
Expand All @@ -66,43 +48,73 @@ pub enum LurkProof<'a> {
},
}

impl<'a> LurkProof<'a> {
#[allow(dead_code)]
fn verify(self) -> Result<bool> {
match self {
Self::Nova {
proof,
public_inputs,
public_outputs,
num_steps,
rc,
lang,
} => {
log::info!("Loading public parameters");
let pp = public_params(rc, std::sync::Arc::new(lang))?;
Ok(proof.verify(&pp, num_steps, &public_inputs, &public_outputs)?)
}
#[cfg(not(target_arch = "wasm32"))]
mod cli {
use crate::cli::{
field_data::FieldData,
paths::cli::{proof_meta_path, proof_path},
};
use anyhow::Result;
use lurk::{
coprocessor::Coprocessor, eval::lang::Coproc, field::LurkField,
public_parameters::public_params,
};
use serde::Serialize;
use std::{fs::File, io::BufReader, io::BufWriter};

use super::{LurkProof, LurkProofMeta, Pallas};

impl<F: LurkField + Serialize> LurkProofMeta<F> {
pub fn persist(&self, id: &str) -> Result<()> {
let fd = &FieldData::wrap::<F, LurkProofMeta<F>>(self)?;
bincode::serialize_into(BufWriter::new(&File::create(proof_meta_path(id))?), fd)?;
Ok(())
}
}

#[allow(dead_code)]
fn print_verification(proof_id: &str, success: bool) {
if success {
println!("✓ Proof \"{proof_id}\" verified");
} else {
println!("✗ Proof \"{proof_id}\" failed on verification");
impl<'a, F: LurkField + Serialize> LurkProof<'a, F>
where
Coproc<F>: Coprocessor<Pallas>,
{
pub fn persist(&self, id: &str) -> Result<()> {
let fd = &FieldData::wrap::<F, LurkProof<'_, F>>(self)?;
bincode::serialize_into(BufWriter::new(&File::create(proof_path(id))?), fd)?;
Ok(())
}

fn print_verification(proof_id: &str, success: bool) {
if success {
println!("✓ Proof \"{proof_id}\" verified");
} else {
println!("✗ Proof \"{proof_id}\" failed on verification");
}
}
}

#[cfg(not(target_arch = "wasm32"))]
pub fn verify_proof(proof_id: &str) -> Result<()> {
use super::paths::proof_path;
use std::{fs::File, io::BufReader};
impl<'a> LurkProof<'a, Pallas> {
fn verify(self) -> Result<bool> {
match self {
Self::Nova {
proof,
public_inputs,
public_outputs,
num_steps,
rc,
lang,
} => {
log::info!("Loading public parameters");
let pp = public_params(rc, std::sync::Arc::new(lang))?;
Ok(proof.verify(&pp, num_steps, &public_inputs, &public_outputs)?)
}
}
}

let file = File::open(proof_path(proof_id))?;
let fd: FieldData = bincode::deserialize_from(BufReader::new(file))?;
let lurk_proof: LurkProof = fd.open()?;
Self::print_verification(proof_id, lurk_proof.verify()?);
Ok(())
pub fn verify_proof<F: LurkField>(proof_id: &str) -> Result<()> {
let file = File::open(proof_path(proof_id))?;
let fd: FieldData = bincode::deserialize_from(BufReader::new(file))?;
let lurk_proof = fd.extract::<F, LurkProof<'_, Pallas>>()?;
Self::print_verification(proof_id, lurk_proof.verify()?);
Ok(())
}
}
}
6 changes: 4 additions & 2 deletions src/cli/mod.rs
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
mod commitment;
mod field_data;
mod lurk_proof;
mod paths;
mod repl;
Expand Down Expand Up @@ -361,7 +363,7 @@ struct VerifyArgs {
/// Parses CLI arguments and continues the program flow accordingly
pub fn parse_and_run() -> Result<()> {
#[cfg(not(target_arch = "wasm32"))]
paths::create_lurk_dirs()?;
paths::cli::create_lurk_dirs()?;

if let Ok(repl_cli) = ReplCli::try_parse() {
repl_cli.run()
Expand All @@ -376,7 +378,7 @@ pub fn parse_and_run() -> Result<()> {
#[cfg(not(target_arch = "wasm32"))]
{
use crate::cli::lurk_proof::LurkProof;
LurkProof::verify_proof(&verify_args.proof_id)?;
LurkProof::verify_proof::<pallas::Scalar>(&verify_args.proof_id)?;
}
Ok(())
}
Expand Down
79 changes: 40 additions & 39 deletions src/cli/paths.rs
Original file line number Diff line number Diff line change
@@ -1,51 +1,52 @@
#[cfg(not(target_arch = "wasm32"))]
use anyhow::Result;
pub mod cli {
use anyhow::Result;

#[cfg(not(target_arch = "wasm32"))]
use std::{
fs,
path::{Path, PathBuf},
};
use std::{
fs,
path::{Path, PathBuf},
};

#[cfg(not(target_arch = "wasm32"))]
fn home_dir() -> PathBuf {
home::home_dir().expect("missing home directory")
}
fn home_dir() -> PathBuf {
home::home_dir().expect("missing home directory")
}

#[cfg(not(target_arch = "wasm32"))]
pub fn lurk_dir() -> PathBuf {
home_dir().join(Path::new(".lurk"))
}
pub fn lurk_dir() -> PathBuf {
home_dir().join(Path::new(".lurk"))
}

#[cfg(not(target_arch = "wasm32"))]
pub fn proofs_dir() -> PathBuf {
lurk_dir().join(Path::new("proofs"))
}
pub fn proofs_dir() -> PathBuf {
lurk_dir().join(Path::new("proofs"))
}

#[cfg(not(target_arch = "wasm32"))]
pub fn lurk_leaf_dirs() -> [PathBuf; 1] {
[proofs_dir()]
}
pub fn commits_dir() -> PathBuf {
lurk_dir().join(Path::new("commits"))
}

#[cfg(not(target_arch = "wasm32"))]
pub fn create_lurk_dirs() -> Result<()> {
for dir in lurk_leaf_dirs() {
fs::create_dir_all(dir)?;
pub fn lurk_leaf_dirs() -> [PathBuf; 2] {
[proofs_dir(), commits_dir()]
}
Ok(())
}

#[cfg(not(target_arch = "wasm32"))]
pub fn proof_path(name: &str) -> PathBuf {
proofs_dir().join(Path::new(name)).with_extension("proof")
}
pub fn create_lurk_dirs() -> Result<()> {
for dir in lurk_leaf_dirs() {
fs::create_dir_all(dir)?;
}
Ok(())
}

#[cfg(not(target_arch = "wasm32"))]
pub fn proof_meta_path(name: &str) -> PathBuf {
proofs_dir().join(Path::new(name)).with_extension("meta")
}
pub fn repl_history() -> PathBuf {
lurk_dir().join(Path::new("repl-history"))
}

#[cfg(not(target_arch = "wasm32"))]
pub fn repl_history() -> PathBuf {
lurk_dir().join(Path::new("repl-history"))
pub fn commitment_path(hash: &str) -> PathBuf {
commits_dir().join(Path::new(hash))
}

pub fn proof_path(name: &str) -> PathBuf {
proofs_dir().join(Path::new(name)).with_extension("proof")
}

pub fn proof_meta_path(name: &str) -> PathBuf {
proofs_dir().join(Path::new(name)).with_extension("meta")
}
}
Loading