Two Missing P's

Arrgghh!!! I hate it when I outsmart myself!! I just finished a three-hour ordeal debugging why my app couldn't connect to my server.

I started by running the raw REST calls through Paw to make sure I was actually calling the REST endpoint correctly and I could create records on the backend.

Since this is an authenticated service I had to first sign-in, so I had to set up a separate configuration for that to get the user's token.

Next I setup the actual call to post the record, putting the user's email and their token in the header and the data for the record I wanted to create into the body (as JSON).

I ran it and everything worked fine. So it should have been as simple as do the same thing in the app. Riiiiiigggghhht.

In the app I'm using AlamoFire to make the REST calls and after successfully logging in I went on to (attempt) to create the new record.

This is what I got in the rails console:

User Load (2.1ms)  SELECT  "users".* FROM "users" WHERE "users"."email" = $1 ORDER BY "users"."id" ASC LIMIT $2  [["email", "test@home.com"], ["LIMIT", 1]]
Completed 400 Bad Request in 19ms (ActiveRecord: 2.1ms)

ActionController::ParameterMissing (param is missing or the value is empty: portfolio):

I know the load of the user is just Devise's before_action kicking in. But why don't I see even the parameters being passed in?

Ok, I thought, I have seen the param missing error before and it means for some reason my payload isn't getting to the server.

So I put in a few debug statements on the Rails side, specifically:

logger.debug "raw post: #{request.raw_post}"
logger.debug "params: #{params.as_json}"
logger.debug "local_params: #{portfolio_params.as_json}"

Note: portfolio_params is my "white list" method for the controller. This resulted in the following output:

User Load (2.5ms)  SELECT  "users".* FROM "users" WHERE "users"."email" = $1 ORDER BY "users"."id" ASC LIMIT $2  [["email", "test@home.com"], ["LIMIT", 1]]
raw post: {"portfolio":{"store_on_server":false, ... }}
params: {"controller"=>"api/v1/portfolios", "action"=>"create", "format"=>"json", "user_email"=>"test@home.com", "user_token"=>"_Csa8xXWzV-sfcazxdwJ"}
Completed 400 Bad Request in 11ms (ActiveRecord: 2.5ms)

Huh? Line 2 confirms my portfolio body is getting to the server, but when I look at the params on line 3, it's no where to be found.

Hmm, so I Google'd how to set the body in Swift 3 using AlamoFire and I found this post:

How to set body on post with AlamoFire

I looked at the example's syntax:

Alamofire.request("http://myserver.com", method: .post, parameters: parameters, encoding: JSONEncoding.default)
    .responseJSON { response in
        print(response)
    }

Here is what I had:

let request = Alamofire.request(urlString, method: .post, parameters: json, encoding: JSONEncoding.default, headers: headerCacheManager.getLatestHeaderFields())

Uhh, mine is the same EXCEPT for the call to add the header fields, but what could be wrong there? Surely the idiot programmer (me), who wrote the getLatestHeaderFields() method didn't mess that up right? I mean I have been successfully using that for months. So no need to check that right!?

Next I printed out the actual JSON on the app side, prior to passing it to AlamoFire:

print("JSON is: \(json.debugDescription)")

It looked right to me. Now what?

So I went old school. First I got the curl representation of what was working from Paw:

curl -X "POST" "http://localhost:3000/api/v1/portfolios" \
     -H "X-User-Token: _Csa8xXWzV-sfcazxdwJ" \
     -H "X-User-Email: test@home.com" \
     -H "Cookie: _pr_session=Uk1vVFUxSmRIS3J2YVN6UUlTRUczbzg4KytIbC9rajdXKzZTUDFiZ0laTnRzaTdCdERDYUFscmE0a05Zd3Y0eGdBV0lYKzNiWFIwOVpjM0Z0c3dpeWVwaWpsZWM2N1Nkb2xWZWRsM2VFazVybDhBd1Jaa2d1OHhQS2ZnQm1pbmo2bzZGTXBSSEh5Y3FtWkZZRnNnUzZFYmVTamVKTTVUUUNHaDFHOTdwb0h3VVNWWVBiS3ptNUlQUHI2Mk9iZXE1aTdnQi9aWk5HbnpzV3dlRXAyNzdFczVrckt5NEwzMk02QmRYSWxyS2IwbndzRFNXNDVsSzdYQlNVd0o4TUd3aC0tQVN0YkV2MWtXTGJyYVl3S0IxaHNLUT09--32ceac68ff051bf41d3437ecdc6b04730c00cc9d" \
     -H "Content-Type: application/json" \
     -H "Accept: application/json" \
     -d $'{
  "name": "Portfolio",
  "uuid": "uuid1"
}'

Pasting this into the console worked fine as well. But how could I see the request as it left the client? That's what I needed.

Eventually I figured out I could print a debug description of the AlamoFire request object which would give me the 'curl' command the app was using.

print("Request: \(request.debugDescription)")

The output of that was:

Request: $ curl -v \
    -X POST \
    -H "Content-Type: aaplication/json" \
    -H "X-User-Token: _Csa8xXWzV-sfcazxdwJ" \
    -H "Accept: aaplication/json" \
    -H "User-Agent: PortfolioRebalancer/1.0 (com.talonstrikesoftware.PortfolioRebalancer; build:1; iOS 10.3.1) Alamofire/4.5.0" \
    -H "Accept-Language: en;q=1.0" \
    -H "X-User-Email: test@home.com" \
    -H "Accept-Encoding: gzip;q=1.0, compress;q=0.5" \
    -d "{\"portfolio\":{\"store_on_server\":false, ...}" \
    "http://localhost:3000/api/v1/portfolios.json"

Do you see the problem? I didn't so I copied this string into a terminal window, executed it, and it failed.

Ok on to comparing each parameter. Do you see the problem now? Yeah, whether it was because it was late, or my monitor font was too small, or the fact I wrote this code months ago, it took me forever to see the problem.

It turns out about a week ago I had refactored the HeaderCacheManager class and I made a typo. What should have been:

headers["Accept"] = "application/json"
headers["Content-Type"] = "application/json"

was actually:

headers["Accept"] = "aaplication/json"
headers["Content-Type"] = "aaplication/json"

I misspelled the word application Doh!!!

Fixing those two p's the server responded like I would have expected:

Started POST "/api/v1/portfolios.json" for 172.17.0.1 at 2017-07-20 21:43:20 +0000
Processing by Api::V1::PortfoliosController#create as JSON
  Parameters: {"portfolio"=>{"store_on_server"=>false, ...}}
  User Load (2.9ms)  SELECT  "users".* FROM "users" WHERE "users"."email" = $1 ORDER BY "users"."id" ASC LIMIT $2  [["email", "test@home.com"], ["LIMIT", 1]]
  Portfolio Load (2.2ms)  SELECT  "portfolios".* FROM "portfolios" WHERE "portfolios"."uuid" = $1 ORDER BY "portfolios"."id" ASC LIMIT $2  [["uuid", "43E48CC3-286E-4CB2-914E-EBEA69707480"], ["LIMIT", 1]]
   (0.3ms)  BEGIN
  User Load (1.9ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = $1 LIMIT $2  [["id", 1], ["LIMIT", 1]]
  SQL (1.8ms)  INSERT INTO "portfolios" [blah, blah ,blah]
   (0.5ms)  COMMIT
Completed 201 Created in 36ms (Views: 0.6ms | ActiveRecord: 9.6ms)

Done!!! In the end that request.debugDescription saved my bacon and I guess I just need to learn to spell better (or type better I'm not sure)

Till next time.

Plan A

The past few weeks I have been working on my new app. It is a stock tracking app and I ran into the issue of how to get live stock prices for free.

To be honest I would pay a small monthly fee for this capability, but the sites I went to that alluded to the fact they might have a service I could pay for didn't publish their prices. They just said "contact us".

I figured that meant I couldn't afford them.

So that sent me down the tried and heavily worn paths such as yahoo finance and screen scraping.

For my application, I need three bits of information, given the equity's symbol.

I needed:

• Title
• Description
• Latest price

In my use case, the user enters the symbol, and I then retrieve the name and description to allow them to confirm this is what they want to use.

Later in the workflow, at specific touch points in the app, I want to retrieve the latest prices for all the equities the user is tracking.

In the end I decided I would choose a hybrid approach.

Most of the solutions I found pointed at using yahoo finance for live stock data. The problem with that was I could not find a way to get the equity's description information.

The only way I could figure to get the description information was lookup the symbol on google and then screen scrape the information.

So my hybrid solution is to screen scrape the information when the user is searching for an equity to track and then once they are tracking it use yahoo finance to pull the live data.

One caveat before I go into the code: I realize that yahoo has been bought out and there is no guarantee the solution I have will work even in the near future. That's one of the reasons I chose to have two routes to the data. But for now this works pretty well.

For this post I wanted to dive into the integration I have for yahoo finance. In a future post I'll talk about the screen scraping approach.

Prerequisites for this solution are: Alamofire

To make this clean and isolated, I created a StockService class like so:

import Foundation
import Alamofire

/**
 A struct to hold a quote for an equity
 */
struct EquityQuote {
    var symbol:String = ""
    var price: Double = 0.0
}

/**
 The Stock Service provides functions to lookup stocks and retrieve their latest prices
 
 - notes:
   see https://developer.yahoo.com/yql/
 */
class StockService {

I created a struct to capture the loaded data and as the comment mentions I lean on yahoo's yql service

The idea is consuming code/classes will call the function on this service object whenever the quotes need to be retrieved. The details of how the service is found (fyi, it's not a singleton) is not germaine to this post. Maybe I should add that to my list of topics to blog about.

At any rate, just assume consumers can get to the service and call the getLatestQuotes method:

func getLatestQuotes(symbols:[String], completionHandler:@escaping (([EquityQuote]) -> Void)) {

The function takes an array of symbols (as strings) and a completion handler function that will return a list of EquityQuote objects.

There are a lot of old posts out there about how to do this, some use YQL and others point to yahoo.finance directly. After a bit of experimentation I came to the conclusion that the YQL solution was more correct for 2017.

Another thing that annoyed me was the url call has to be url encoded (i.e. spaces turned into their safe ASCII equivalents, etc). A lot of the examples you will find hard code the conversion in their strings.

To me this made it hard to look at and debug. So I chose to write as I would in the yql console and right before making url encode it.

So I defined constants for the things in the query that don't change. More specifically the host, the query prefix and the query suffix:

let QUOTE_QUERY_HOST = "http://query.yahooapis.com/v1/public/yql"
let QUOTE_QUERY_PREFIX = "?q=select * from yahoo.finance.quotes where symbol in ( "
let QUOTE_QUERY_SUFFIX = " )&format=json&env=store://datatables.org/alltableswithkeys&callback="

Next I convert the symbols array into a single string and then cobble together the entire url.

var symbolsString = symbols.reduce("") {text, value in "\(text)\"\(value)\", "}
symbolsString = symbolsString.truncate(by:2)

let finalQuery = "\(QUOTE_QUERY_PREFIX)\(symbolsString)\(QUOTE_QUERY_SUFFIX)"
let encodedQuery = finalQuery.urlEncode()
let urlToCall = "\(QUOTE_QUERY_HOST)\(encodedQuery)"

Note the call to finalQuery.urlEncode(). This is a function I put in a String extension to encode all the characters that need to be encoded to go across the wire. I won't include that here, but if anyone is interested let me know.

Now that I have a safe encoded url, it is time to make the call via Alamofire:

let request = Alamofire.request(urlToCall, method: .get, parameters: nil, encoding: JSONEncoding.default, headers:nil)
    request.responseJSON() { response in
        let result = response.result
        switch result {

As you can see it is a "get" call and I expect JSON back.

If I get a success then it's time to parse it. This part of the code is UGLY!! and I need to go back and use something like "swiftyJSON" to clean up all the checks. But, at the time, I was doing a lot of experimenting to get this to work and it was easier to just peel one layer off at a time.

case .success:
    if let value = result.value as? [String: Any] {
         if let query = value["query"] as? [String: Any] {
              if let results = query["results"] as? [String: Any]  {
                   if let quotes = results["quote"] as? [[String: Any]] {
                        var equityQuotes = [EquityQuote]()
                        for quote in quotes {
                            let symbol = quote["symbol"]
                            let lastTradePrice = quote["LastTradePriceOnly"]
                            if let symbol = symbol as? String, let priceText = lastTradePrice as? String, let price = Double(priceText) {
                                 let equityQuote = EquityQuote(symbol: symbol, price: price)
                                 equityQuotes.append(equityQuote)
                            }
                        }
                    completionHandler(equityQuotes)
                    return
                }
            }
        }
    }
    completionHandler([EquityQuote]())
case let .failure(error):
    print("\(error)")
}

What is going on here is I basically am diving into the returned JSON until I get to the "quote" dictionary. I then pull out the information I need to build an EquityQuote object and if I have them, I construct the object and throw it into the array I will return. Finally after processing all of that, I call the completion handler to notify the calling code we are done.

Things I still have to do is deal better with error conditions. It may be that I change the method signature to take a failure handler as well.

Note, if I get nothing from the call I still return an empty array to the caller. I'm not sure in what cases this would happen but it seemed like the right thing to do in order to continue the code flow.

I could have just returned an array of doubles, but I figured by returning these EquityQuote objects the calling code could inspect the symbol of each one and match it up with the symbols it cares about. Again, it seemed safer.

That's it, so far it is working pretty well. My next post will talk about the screen scraping approach I took. Till next time.

Back in the Game

Mac update and

Its been a while since I last made a post so it is about time.

I wrote this back in January, I just never posted it. First, my 2016 MacBook Pro (MBP) saga update.

I was having so much trouble converting from using my MBP in a mobile configuration back to what I call my "desktop" configuration (which is clamshell mode with two external 4k moniters attached) that I came real close to taking it back.

It would rarely make the conversion from internal GPU to discrete GPU without either crashing or just losing it's mind. I could never count on the transition to work. I could usually count on it to NOT work.

But with the update to macOS 10.12.3 (which hinted it fixed a GPU switching problem) it has become much more stable. I still don't go from "mobile" (clamshell closed) mode to "desktop" mode without first opening the clamshell prior to connecting the monitors. But it is much better now.

It's not perfect, but now I have a 50/50 chance of waking my computer up by just hitting a few keys on the keyboard.

The battery situation still isn't great but seems to have gotten a little better with the macOS update and the latest version of Chrome.

So in the end it took over two months but I am at least satisfied with my purchase. Here is a picture of my setup:

On the development front, I have started two new apps. One is a financial app for iOS and the other is my first game using Unity. Both are a long way away but I am excited about the endeavors.

If I had to guess, I would expect the finance app to take about 6 months and the game may never be released (if I am truly honest with my time commitments).

Now to the more interesting stuff. My side web app project.

This project is a Ruby on Rails backend supporting (right now) a mobile app via a REST api. In order for us to release the product though, we also will need a web app.

I built a basic admin portal for the database and, several months ago, built the first part of the web client using React and the 'react-rails' gem.

As time went by, our design team turned to focus on the mobile side and as (often happens in startups) the web design languished as we discovered new workflows and data structures needed for the app.

That gave me time to read and study how best to build the web front-end based on React for a RoR backend.

After much studying I decided I should use webpack and build the React client with a true Javascript tool chain.

I first started by rolling my own before I found the 'react-on-rails' gem. This has an opinionated way of integrating a Javascript toolchain into a RoR application.

It looks good but I find the example's a little hard to grok. Whether that is my lack of knowledge of RoR or Node I'm not sure. I did read yesterday that RoR 5.1 will have Webpack support baked in, so I am looking forward to that.

I think this is where I will stop today. Till next time.