Monday, April 27, 2015

Week 1 With Ensembles

My goal this week was to build a stack to have Core Data on the local device and use Ensembles to sync the data to iCloud.  I’m not finished yet.

Progress
After reading 100 pages in the Ensembles book I felt I was ready to go.  I quickly learned I needed an example to make much progress.  I opened up the Idiomatic Example application and worked through it.  I’m a little stuck on the use of the container id.  It is unclear to me what I am supposed to use for the ubiquity container Identifier.  Do I use nil as in the idiomatic example, do I use some arbitrary String as in the simple sync example, or do I use the ubiquity container id as I would with straight CoreData?  I think I am going to try the standard ubiquity container id as I would do if I was using CoreData sync.

A second issue I ran into was how to implement persistentStoreEnsemble(ensemble: CDEPersistentStoreEnsemble!, globalIdentifiersForManagedObjects objects: [AnyObject]!)
in Swift. I’m not sure what I will do with this yet.

I posted the problem I was talking about in last weeks post with my CoreData solution to the Apple CoreData forums.  I’m not sure whether no one understood what I was asking or no one has an answer.  But at any rate, its been four days since my post and NO ONE has responded, although I know it has been looked at.  Hmm, I wonder if there is a problem here :-)

Looking back at my CoreData+Sync solution it appears that I might have a threading issue.  I’ve also decided I am going to relook at Parse or CloudKit.  So it looks like I have three test apps in play right now:

CoreData + iCloud Sync - My plan here is to relook at the threading concerns to see if maybe that is the real culprit here.

CoreData + Ensembles - For this one I need to put some more code down before I can start testing.

Parse/CloudKit - I don’t have a test project for this yet, but having worked with it in the past it seems like it won’t be too bad.  I just need to find time to do it.

Reading
I’m over halfway through Spring in Action now.  I have a test app working with both login and logout and a few protected and unprotected web pages.  I just finished the section on persistence.  I’ll probably go with a standard SQL solution for that as I am most comfortable with that type of a solution.

News That Caught My Eye
I watched the keynote from RubyConf 2015 this week.  They introduced Rails 5.  It looks like the Rails community is still thriving.  The big question in my mind is do I continue to use Rails and bring my web apps into the new version, or do I swing back to my Java roots and convert to Spring solutions.  

Quick Looks

I looked briefly at Realm for my iOS applications this week.  It is very interesting and I really want to use it, but I think I would need to stand-up my own server for cross device syncing.  I think that is where I ultimately will end up for all my solutions, but right now I need to get the current test apps done.   

Sunday, April 19, 2015

More Questions Than Answers

Progress
I have not been able to solve the problem I was dealing this last week getting this error while syncing:

CoreData: Ubiquity:  Librarian returned a serious error for starting downloads Error Domain=BRCloudDocsErrorDomain Code=5 "The operation couldn’t be completed. (BRCloudDocsErrorDomain error 5 - No document at URL)"

What I did find out though was that when I deployed my test app on two devices using the same iCloud account MOST of the changes would sync.  But a few never did. If I deleted them while connected to iCloud, then migrated to my local store, then migrated that store back to iCloud they (and the errors) would reappear.  

I’m suspecting that the problem is with this error I am seeing.  I’ve seen one post recently on the developer forums that asked questions like “Is your store read only, do you have the correct permissions to the store, etc”. But since I have followed the Apple examples and the Core Data + iCloud API is so opaque I have no idea on where to look or how to fix it.  The bottom line is I am becoming more and more suspect that Core Data with iCloud still isn’t reliable for production work.  It has been improved, in the past it would just fail to sync altogether, but now it “mostly” works.  But that’s not good enough for me.  I found this post from late year and I think the blogger nails it on the head about the issues:


But I haven’t totally given up yet.  I picked up the Ensembles book and my current direction is to see if I can use Core Data on the local device using a local store and hook into the Ensembles framework to do the syncing with the cloud.  That feels like a happy medium and meets my original persistence requirements.  If that doesn’t work then I am left with going down the path the post I just mentioned, goes.  I think in the end I would like to go with my own REST service, but in the interest of getting my app out the door I’m willing to spend a couple of more weeks exploring Core Data with Ensembles.  So that is my current direction.

I did solve the lockup problem I was having with my deduplication algorithm.  It turned out to be a threading issue with the callback method on my save routine (which I posted last week).  To switch it I had to change the call to the callback method to be an async operation instead of sync.  I did update the code in that post. I guess the sync was locking up the async call it was in and since the callback needed to access the store and the async it was being called from also wanted to operate on the store the two were contending for the same resource and locked everything up.

Sorry no code to post this week.  Hopefully, I will make progress with Ensembles and be able to post code next week.  My goal is to post as much working code as I can so that others can either learn from my mistakes, tell me what I am doing wrong, or use it as a springboard to bettering their code.

Reading
I’m halfway through reading Spring in Action from Manning Publishing.  I was able to get the test app deployed this week.  Can now login, can’t successfully log out but that will come.

News That Caught My Eye
WWDC tickets went out this week.  I can’t imagine what Apple will announce this year.  Last year was so astounding with WatchKit and Swift.

Quick Looks

I briefly looked at FMDB.  I don’t think I need to go to that extreme to solve my persistent issues.  I think before I went that direction I would switch over to the Parse framework.  Come to think of it that is still a possibility.  I also did some work with Unity 5.  That is such a joy of an API and IDE to work with.  

Sunday, April 12, 2015

Two Steps Back

I’ve decided to break my blog down into four sections so that I cover the topics I intend to each week.  First will be “Progress” where I highlight the things I learned this week. Next will be “Reading” where I talk about what I am currently studying.  Following that will be “Quick Looks” where I talk about some technologies I briefly researched during the week.  Finally I will have “News” where I will talk about news that caught my eye this week.  So without further ado, here goes:


Progress
I didn’t make as much progress as I had hoped this week.  I did add Marcus Zarra’s CoreDataStack design to my project.  It was pretty straight forward.  Here is the initialization in Swift:
   init(dbName:String, syncToCloud:Bool, completion: ((Void) -> Void)? ) {  
     modelName = dbName  
     // Get the Model  
     let bundle = NSBundle.mainBundle()  
     let modelURL = bundle.URLForResource(dbName, withExtension: "momd")  
     let _model = NSManagedObjectModel(contentsOfURL: modelURL!)  
     assert(_model != nil, "Could not retrieve model at URL \(modelURL)")  
     model = _model!  
     // Build the persistent store coordinator  
     psc = NSPersistentStoreCoordinator(managedObjectModel: model)  
     // Set up the main MOC  
     mainQueueMOC = NSManagedObjectContext(concurrencyType: .MainQueueConcurrencyType)  
     // Create the private MOC, this is the parent context of the mainQueueMOC and it  
     // owns the persistent store.  
     privateMOC = NSManagedObjectContext(concurrencyType: .PrivateQueueConcurrencyType)  
     privateMOC.persistentStoreCoordinator = psc  
     // Make the main MOC a child of the private MOC  
     mainQueueMOC.parentContext = privateMOC  
     // Attach the basic life cycle listeners  
     attachInternalListeners()  
     dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)) {  
       // Get the proper options dictionary and storeURL based on the state  
       // of our sync flag.  
       var storeURL:NSURL?  
       var options:[NSObject: AnyObject]?  
       if (syncToCloud) {  
         storeURL = self.cloudStoreURL  
         if (self.rebuildFromCloudStore) {  
           options = self.cloudStoreOptionsWithRebuild  
         }  
         else {  
           options = self.cloudStoreOptions  
         }  
       }  
       else {  
         options = self.localStoreOptions  
         storeURL = self.localStoreURL  
       }  
       var error: NSError? = nil  
       self.store = self.psc.addPersistentStoreWithType(NSSQLiteStoreType, configuration: nil, URL: storeURL!, options: options!, error: &error)  
       assert(self.store != nil, "Could not add store URL \(storeURL)")  
       if let _completion = completion {  
         dispatch_async(dispatch_get_main_queue(),_completion)  
       }  
     }  
   }  


Where I got tripped up is in running the deduplication algorithm.  It worked but after migrating the store to iCloud then running the deduplication code and standing up a new NSFetchedResultsController I get the following error:


CoreData: Ubiquity:  Librarian returned a serious error for starting downloads Error Domain=BRCloudDocsErrorDomain Code=5 "The operation couldn’t be completed. (BRCloudDocsErrorDomain error 5 - No document at URL)"


Doing a few google searches I found that others are also seeing this error, but so far there is no answer.  I’m not sure this is really the error that is causing all the issues.  What I am seeing in my test app is if I include the deduplication code in the code path after the store is migrated (whether that be to the cloud or to local store) my app hangs right when I try to update the fetched results for my UITableView.  And yes, I am on the main queue when it happens so I know that is not the problem.


I’m also still running across folks giving up on CoreData and iCloud syncing so my decision to continue this investigation seems a little shakey.  I just wished I knew what they were moving to, since the problem of sync’d data seems like a required feature in today’s apps.  I do not want to stand up my own server to do this (although I am entirely capable of doing it) because I don’t want to incur the recurring cost for an app that might not be able to pay the server costs.


Reading
I am currently reading Spring in Action from Manning Publishing.  I’m about a third of the way through and it appears to be very thorough.  I need to work on a test app to apply what I am learning


News That Caught My Eye
Well the Apple Watch was opened for pre-order this past Friday.  I was up with the intent of buying one, but in the end I decided against it. At least for now.  I think I can get buy with the simulator as the supply chain gets filled and the first real users start reporting on it.


Quick Looks

I intended to look at React this past week, but ran out of cycles.  Maybe next week.

Sunday, April 5, 2015

A Light?

A few days ago I was ready to give up.  I was so depressed over trying to get Core Data sync working.  My personal feeling right now is LEARNING CORE DATA IS A CHORE!!!  I guess at this point I know enough to be dangerous.  Swapping in and out of cloud syncing almost seems to be impossible.  Not to mention adding Swift and MagicalRecord to the mix just made it even more challenging.  So continuing on:


After contemplating tossing all my devices in the trash and going fishing, I decided to try the simplest thing that could work.  I created a new test app and decided to just do the most minimalist Core Data (CD) app I could and add syncing in as I went.  I think at this point there may be a light at the end of this very narrow, long and twisty tunnel. I found this blog post which I thought was really interesting:

Martin Craft's Core Data Stack

I really like Marcus’ assessment of the state of information about getting a proper CD stack setup.  He absolutely nailed it, it is all over the place and there are A LOT of red herrings. I also found this link to a CD book that looked promising (but I have not read it yet):

Tim Roadly Core Data Book


But there appears to be a new version being worked on using Swift.  So if I do decide to buy it I’ll wait for that version.


One side issue I ran into was I had attached listeners to all the events that CD throws and in my listeners I was logging them using NSLog and trying to print out the notification using Swift’s inline string parameterization (I forget the technical term for this and am too lazy to look it up right now).  It worked fine on the simulator but when I moved to a real device, I got BAD_EXEC errors.  Switching to use “println” instead seemed to solve the problem.  So I have added that gem of information to my list of items to try when Swift seems to be misbehaving.


Another thing that helped was to mark my class that was handling the CD stack with “@objc” this seemed to calm down a lot of errors I was dealing with when listening for events coming from CD itself.


So for review here is my original list of requirements:


  1. Make an app agnostic class to manage a Core Data stack in Swift
  2. Be able to switch from a local store to a cloud store
  3. Be able to switch from a cloud store back to a local store
  4. Be able to remove the cloud store all together
  5. Be able to blow away local changes and rebuild from the cloud store
  6. A simple way to hook into CD notifications
  7. Support schema migrations


My status so far is this:


Item 1 - Done.  I take a model name in the initializer for the class and then build everything else from there.


Item 2 - Partially done.  I can now make the switch to iCloud from a local store.  What doesn’t work is after doing this the first time then switching back to a local store and then switching back to the cloud store. I end up with duplicate records.  That makes sense to me as when I switch to the local store the device loses knowledge that those items existed in the cloud.  When going back to the cloud the device dutifully merges it’s objects with those in the already existing cloud store so I end up with duplicates.  This is the step in Apple’s iCloud with Core Data store talking about deduplication of records.  


1:   func switchToCloudStore(completion: ((Void) -> Void)?) {  
2:      let cloudOptions = cloudStoreOptions  
3:      let storeURL = cloudStoreURL  
4:      if (storeURL == nil) {  
5:        println("Could not switch to store because store URL not available")  
6:        return  
7:      }  
8:      var error: NSError?  
9:       let newStore = psc.migratePersistentStore(store!, toURL: storeURL!, options: cloudOptions, withType: NSSQLiteStoreType, error: &error)  
10:       if let _newStore = newStore {  
11:        println("Cloud store URL is: \(psc.URLForPersistentStore(_newStore))")  
12:        self.store = _newStore  
13:        context.reset()  
14:        if let _completion = completion {  
15:          _completion()  
16:        }  
17:      }  
18:      else if let _error = error {  
19:        println("Error in migrating to cloud store: \(_error)")  
20:      }  
21:      else {  
22:        println("Returned newStore was nil when trying to migrate to cloud store")  
23:      }  
24:    }  

Item 3 - Partially done.  This has the same problem as item 2 where I move back to a local store (that already existed and had records in it).  I end up with the records from the cloud store as well as the set of records that were already existing in the local store.


1:    func switchToLocalStore(completion: ((Void) -> Void)?) {  
2:      var localOptions = localStoreOptions  
3:      localOptions.updateValue(true, forKey: NSPersistentStoreRemoveUbiquitousMetadataOption)  
4:      var error : NSErrorPointer = nil  
5:      if let _newStore = psc.migratePersistentStore(store!, toURL: localStoreURL, options: localOptions, withType: NSSQLiteStoreType, error: error) {  
6:        self.store = _newStore  
7:        context.reset()  
8:        if let _completion = completion {  
9:          _completion()  
10:        }  
11:      }  
12:    }  

Item 4 -  Partially works - This appears to work, but it appears I need to remove the store from all devices before it completely goes away.  This doesn’t totally surprise me, but it is still a little unnerving.  I can live with it though.


1:    func resetStore(completion: ((Void) -> Void)?) {  
2:      let storeURL = cloudStoreURL  
3:      if (storeURL == nil) {  
4:        println("Could not reset store because store URL not available")  
5:        return  
6:      }  
7:       let options = [NSPersistentStoreUbiquitousContentNameKey: "\(modelName)CloudStore"]  
8:       var error : NSErrorPointer = nil  
9:      let result = NSPersistentStoreCoordinator.removeUbiquitousContentAndPersistentStoreAtURL(storeURL!, options: options, error: error)  
10:      if (result) {  
11:        if let _completion = completion {  
12:          _completion()  
13:        }  
14:      }  
15:    }  

Item 5 - I haven’t started on this yet, but it seems straight forward.


Item 6 - Done.  I implemented two methods on my class, attachInternalListeners() and attachListenerForChangeEvents(AnyObject,Selector).  The first method is internal for the stack manager to do the right things when changes come in.  The second method is for external consumers of the stack manager so they can attach to change events and do the right thing when those happen.  It is intended for things like a table view controller to get updates when the underlying store changes due to the ubiquitous store changing.  I thought my view controller would automatically update because it has a NSFetchedResultsController but so far I haven’t got that to work.  More work to be done here.


Item 7 - I don’t think there is anything to do on this one.  Reading Apple’s documentation it sounds like syncing will be disabled on a device until it gets to the current version in iCloud.  Now, how all this works depending on the sync state of the device is anyone’s guess.  I suspect I will have to use items 4 or 5 to fix any issues
So right now, I feel like I have made progress.  I still need to deal with threading issues so my code is a “good citizen” of the CPU.  I’m planning on basically setting my contexts up like Marcus Zarra did in his article (this is mostly how I think MagicalRecord worked and how I tried to roll my own in the past)


The other issue I need to deal with is the deduplication of records as I transition to and from the cloud store.  This might be a little more harder than I thought.  My plan is to go back and look at how Apple suggests dealing with that.


Once I am done with all that I’ll probably end up posting the code somewhere so everybody can tell me what I did was wrong or possibly learn something from my ordeal.

Till next time, hoping for more progress.