Consider removing (non-interior) mutability?

[madsmtm]

So... I've kinda considered this problem many times, #265 contains some of the previous discussion and my head contains even more, but I feel like we're still not at an optimum.

Currently, objc2 contains a lot of complex machinery to make classes able to opt-in to being &mut, and icrate uses this on NSMutableString, NSMutableAttributedString, NSMutableArray<T>, NSMutableSet<T>, NSMutableDictionary<K, V> and NSEnumerator<T> to make accessing the inner state of those types safe.

A few examples of current issues in icrate with this approach:

• NSMutableAttributedString::mutableString must return a reference bound to the attributed string, as otherwise you'd be able to create two Id<NSMutableString> to the same string. But this doesn't work with Id<T>, so we'd need a lifetime annotation in there somehow (or maybe resort to autorelease pools).
• NSTreeNode::mutableChildNodes has to remain unsafe, as we have no way of preventing calling that multiple times. Or maybe NSTreeNode also has to be mutable? But that in turn would push NSTreeController to being mutable, which in turn needs us to change NSEditor.
• While NSThread::threadDictionary has to remain unsafe because of thread-safety concerns, it cannot even be used from the owning thread safely.
• In user code, storing one of these mutable objects in a custom object needs to be done using a RefCell.

So while &mut does work for the select few types we've introduced it on, my main problem is that this is not enough! &mut is infectious, and we need to introduce it consistently, everywhere!

And that, in turn, is just not feasible (time-wise, Apple has a lot of API surface)! Especially so with our goal of making header-translator a user-accessible tool, which means user-controlled code, and in turn means users would also have to add extra configuration to handle this for their mutable types, which is a tough ask.

Honestly, maybe we should abandon this endeavour, and instead go down the beaten path Swift has taken; do not try to do anything about mutability, i.e. mark NSString, NSMutableString and so on as InteriorMutable, and have all methods accept &self?

---

All of this is not to say that &mut doesn't have advantages; it does, very much so! And removing it does have several performance-related disadvantages, namely:

• NSString::as_str would have to be unsafe, since the internal storage for the string could now be changed through &self. This has a perf-impact on the impl Display for NSString, as the &mut Formatter<'_> in there could contain a &NSMutableString internally that pointed to the same string, and hence we'll probably need some sort of allocation.
• NSData::bytes would have to be unsafe, which in turn means that the nice impls of Deref, Index, Write and so would be gone.
• Other methods using NS_RETURNS_INNER_POINTER would have to be unsafe.
• Getters on collections would have to retain the object (this is also what they do in Swift), whereas before we could avoid this since they're guaranteed to point into internal storage, and we knew we weren't changing that storage.
• Iterators would have to retain the element being iterated over, to avoid problems if the iterator is invalidated - see here for an example of the issue.
  • Even then, are they sound? Is the fast iterator mutation detection enough? What about if the iterator is passed in and out of an autorelease pool?
  • Will have to be researched (and fuzzed), but a safe (inefficient) fallback is always to just use array.objectAtIndex(i).
• NSString, NSMutableString, NSArray, and so on would loose their Send + Sync-ness.

Some of these performance implications could be resolved by making extra unsafe methods with a safety precondition that the object is not mutated for the lifetime of the return value, although that is still a worse user-experience.

[madsmtm]

I feel like I need to know more about other frameworks before I'd be ready to remove it from objc2, but if we do that, that would have huge benefits too, partially in implementation, but mostly in teachability; Id<T> would be just like Arc<T>, no "but sometimes like Box<T>" caveat, ClassType::Mutability could be removed (with MainThreadOnly being inferred from the Super type, or explicitly requested with a const), ClassType::retain could be unconditionally available (and be moved to Message), _mut methods could be removed, ivars could be moved back to being directly on the type as methods (since we no longer need to generate _mut methods, which was difficult for discoverability), and so on.

Also beneficial: The difficult mutability considerations in future work like #562 and #474 would become much easier.

[madsmtm]

Note that it should be possible to re-introduce &mut downstream (maybe in a crate like cacao, if they felt that to be necessary), one approach could be something like the following.

#[repr(transparent)]
pub struct Owned<T>(Id<T>);

impl Owned<T> {
    // Safety: The id must be the only reference to the object.
    pub unsafe fn new(id: Id<T>) -> Self {
        Self(id)
    }
}

impl Owned<NSString> {
    pub fn to_str<'r, 's: 'r, 'p: 'r>(&'s self, pool: AutoreleasePool<'p>) -> &'r str {
        // SAFETY: The NSString is not mutated for the lifetime of the
        // returned string.
        //
        // We know this because the only methods that mutate the string is in
        // `NSMutableString`, and those take `&mut self`.
        unsafe { self.0.to_str(pool) }
    }

    // Safety: The string must not be retained past the lifetime of `&self`,
    // and may not be mutated through `NSMutableString`.
    pub unsafe fn as_ref(&self) -> &NSString {
        &self.0
    }
}

impl Deref for Owned<NSMutableString> {
    type Target = Owned<NSString>;
    
    fn deref(&self) -> &Owned<NSString> {
        // SAFETY: `Owned` is `#[repr(transparent)]`, and `NSString` is a subclass of `NSMutableString`
        unsafe { mem::transmute(self) }
    }
}
impl DerefMut for Owned<NSMutableString> { ... }

impl Owned<NSMutableString> {
    pub fn replaceCharactersInRange_withString(&mut self, range: NSRange, a_string: &NSString) {
        // Safe to mutate the string because we take `&mut self`
        self.0.replaceCharactersInRange_withString(range, a_string);
    }

    // other `&mut self` methods
}

[madsmtm]

CC @silvanshade, @simlay and @yury.

I know that you are not necessarily familiar with objc2's handling of mutability, but I would like to get some input from you; what are some use-cases for &mut in Objective-C APIs that you've encountered in the past? Do you know of frameworks where single-ownership is very ingrained, or perhaps even central to the framework? Do you know of classes where a retain would be a bad idea?

[silvanshade]

@madsmtm Yeah, this seems like a complicated and subtle issue and to be honest I'm not really sure what the right approach is in the long run.

I think it's probably impossible though to be completely correct in our handling of mutability.

For one, like you suggest, the surface area for the APIs is too large to reasonably cover given the resources we have, even if that wasn't a problem (and we had better tools), there's still the problem of the documentation and headers not being perfect in this regard.

I also worry that icrate and the related libraries might become too difficult to use at some point in the attempt to reach the goal of perfect soundness.

Having to give up on handling mutability accurately is kind of disappointing though.

On the other hand, if you look at the current state of interfacing Rust with Objective-C, icrate is a huge step up even if it (by dropping the goal of accurate handling of mutability) it wouldn't be perfectly safe. It would still be massive improvement over the current status quo.

And I guess even if you did remove some of the mutability-related machinery now to make things a little easier to handle overall, there would still be the possibility of introducing a more accurate interface later on if a suitable approach is found.

I don't have a lot of experience programming with Objective-C or Apple frameworks in general so I don't have a lot to add about specific instances where this might be a problem.

I think that just from the examples I worked on it would be a little daunting to have to have gone through and figured out where everything should have been mutable in the API and fixed that though. To me it was an acceptable trade off to work with an unsafe interface in order to have practically usable bindings. The alternative would have been rolling the bindings by hand which would have undoubtably been more unsafe.

In any case, which ever path you choose here is fine with me.

By the way, regarding the clang-importer work, I just want to mention that I haven't forgotten about that, just need a little break to work on something else, since I started to feel a bit burned out after all the llvmup work. But I've since made progress on libraries for another project which are relevant for the CLI part of that so I plan to get back to that stuff soon.

[yury]

I'm working on streaming app. Main challenge is to maintain battery life. So in cidre I push to efficient calls as much as possible.

I model via &mut self all interfaces for now. No cons for now, as user understands original objc-c API. Anyway cidre is more experimental, not production level (yet).

Most interesting mut/throws/safe Apple API is AVCapture. Many methods just throws bc device is not locked for configuration. I tried to model that around with ConfigLockGuard. But I don't think that is possible with just header translation without "rust API notes".

[madsmtm]

@silvanshade, in general good points, agree that loosing mutability and the safety you get from that is unfortunate, but I think the end result will end up being safer (if not only because users will understand it better, and be better equipped to determine when their unsafe operations are safe).

By the way, regarding the clang-importer work, I just want to mention that I haven't forgotten about that, just need a little break to work on something else, since I started to feel a bit burned out after all the llvmup work. But I've since made progress on libraries for another project which are relevant for the CLI part of that so I plan to get back to that stuff soon.

No worries at all, take all the time you need, am in no rush here! Personally, I've been doing all sorts of other stuff to avoid having to do that, it really is a daunting task!

I model via &mut self all interfaces for now. No cons for now, as user understands original objc-c API. Anyway cidre is more experimental, not production level (yet).

Yeah, though as we discussed, that isn't sound, so that wouldn't really work for objc2.

Most interesting mut/throws/safe Apple API is AVCapture.

Thanks for the example, I'll try to have a closer look at the AVFoundation framework!

[madsmtm]

I looked at all usage of &mut self in your av module, I think there's only really one place where mutability might be essential, and that's mutableAudioBufferList, but I think that'll have to return a raw pointer in any (generated) interface anyhow, as its size depends on the first field of it.

ConfigLockGuard

Agree that this can't be modelled automatically by icrate, but it should be fairly trivial for a crate like cacao or cidre to add a wrapper on top, as you've done.

---

I'll investigate how efficient it is possible to make iterators when we have interior mutability, but still interested in further examples of where mutability might be beneficial to an Objective-C framework (especially methods that return &mut Something). Or maybe examples of Objective-C (or C) frameworks that use -fbounds-safety in a way that allows us to ensure single ownership?

[madsmtm]

I tried to use objc2-metal in wgpu today, and was impacted quite severely by the fact that &ProtocolObject<dyn MTL*> can't be retained, since we don't know if it the class the protocol object came from was originally mutable.

We could fix this by adding : IsRetainable to the metal traits, just noting that it's a problem that wouldn't be there if we didn't try to support mutability.

[madsmtm]

So, it took a while, there was a lot of code that I needed to wrangle this assumption out from (it goes all the way back to the original objc-foundation), see the backreferences above.

But it's done now, yay!