NSPredicate: an old API with new surprises

Douglas Hill •

Recently I was working with NSPredicate — an API that’s been around since Mac OS X Tiger was released in 2005 — and a situation that looked fairly basic wasn’t working as I expected.

I’ve been implementing support for Apple Shortcuts in my reading app so users can create automated workflows. I noticed certain property-based article queries using EntityPropertyQuery weren’t returning the expected number of articles. I had fourteen articles saved on the iPad simulator. Four of these articles were written by me. However when I searched for articles where the author was not “Douglas Hill”, there were only two results instead of the expected ten.

It was clear that articles were not being included where the article’s author was not set. In other words, when the author property was nil. (I’ll mix the terms nil and null in this article because these represent the same concept with different names in different software stacks.)

Tracking down the problem

Firstly let’s consider the most basic test:

let maybeString: String? = nil
let condition = maybeString != "test"

The expectation is that condition would be true in this case. If we’d used ==, then the result would clearly be false. However this uses != so we expect the opposite. Good news: this is indeed how it works!

Secondly I ran a quick test in a playground using NSPredicate in a simple situation:

class MyObject: NSObject {
    @objc var author: String?
    init(author: String?) {
        self.author = author
    }
}

let array = [
    MyObject(author: "Douglas Hill"),
    MyObject(author: "Someone else"),
    MyObject(author: nil),
]

(array as NSArray).filtered(using: NSPredicate(format: "author != %@", "Douglas Hill"))
// [{NSObject, author "Someone else"}, {NSObject, nil}]

This test showed that filtering for objects where the author was not “Douglas Hill” did include objects where the author was nil. This is the behaviour I’d expect.

At this point I was strongly suspecting this was related to the SQLite store that my Core Data stack is using. I’m sure SQL veterans know the answer already.

Thirdly, I did some debugging with my Core Data store without involving Shortcuts, and saw the same as what I saw with Shortcuts: filtering for an attribute not being equal to some value would not include objects where that attribute was nil.

I enabled -com.apple.CoreData.SQLDebug 3 and this showed that the SQL commands being generated were straightforward. This predicate: author != "Douglas Hill" would add this to the SQL SELECT command:

WHERE  t0.ZAUTHOR <> ?

Where the value of the ? is:

SQLite bind[0] = "Douglas Hill"

I’ve never worked with SQL directly, only with it as an implementation detail (and performance detail) of Core Data. At this point my hypothesis was that this handling of null was just how SQL works.

Sadly, SQL doesn’t seem to be a free and open standard where you can easily read the reference/specification to verify a detail like this. I did some research online and second-hand sources supported my hypothesis. NULL is not considered equal or unequal to anything in SQL, or in other words, comparisons with null are neither true nor false.

This comment by jsumrall on a Stack Overflow question sums it up well:

It should also be noted that because != only evaluates for values, doing something like WHERE MyColumn != 'somevalue' will not return the NULL records.

What would a user expect?

From a programmer’s point of view, I wouldn’t say either way to handle null is unequivocally better. However I‘d expect consistency from NSPredicate. The surprising thing to me is that Core Data doesn’t smooth over this behaviour of SQL in order to match how comparisons usually work on Apple’s software stacks.

From a user’s point of view, I think the situation is different. Users won’t be as keenly aware of the concept of null. There is a good chance they think of null and an empty string as being the same. Since my queries will be exposed to users through Shortcuts, I think it’s more expected that filtering for items with a property not equal to some value should include items where that property is null.

Implementing better behaviour

It’s easy to smooth over this quirk ourselves. When setting up a predicate for a Core Data SQLite store with a condition of being not equal to some value. Don’t set up the predicate like this:

NSPredicate(format: "%K != %@", stringKey, nonNilValue)

Instead we also check for equality with nil/null, setting up the predicate like this:

NSPredicate(format: "%K != %@ OR %K == NIL", stringKey, nonNilValue, stringKey)

In practice, here’s what that looks like as a convenience extension on NotEqualToComparator from Apple’s App Intents framework (the Shortcuts API):

private extension NotEqualToComparator<EntityProperty<String?>, String?, NSPredicate> {
    /// Creates a comparator for case- and diacritic-insensitive matching of an optional string property using an NSPredicate for Core Data objects. (My objects are articles.)
    convenience init(keyPath: KeyPath<ArticleEntity, EntityProperty<String?>>) {
        // Maps from Swift key paths to string keys.
        let stringKey = Article.stringKey(from: keyPath)
        self.init() { value in
            if let value {
                return NSPredicate(format: "%K !=[cd] %@ OR %K == NIL", stringKey, value, stringKey)
            } else {
                // Ignore this branch for now since Shortcuts doesn’t have any UI that lets a nil value be passed here. My actual code is slightly different due to an interesting reason, but that’s not the topic of this article.
            }
        }
    }
}

Summary