no comments 
Andre
For a recent Rails project, I had to use PostgreSQL instead of the standard MySQL for the database. Setting up Postgres on Mac OS 10.5 has some quirks, which I will share with you here.
- There are three primary ways to get PostGRES onto Leopard: a pre-built binary, MacPorts, and source. My choice was to build with source. There are several good posts on building from source:
I do not recommend creating a separate user to run postgres. Doing so clutters your startup screen with another user, and requires you to su to that user to start the database. Instead:
1) install Postgres as root
2) after install, from the command line: chown -R andre /usr/local/pgsql/data. The -r flag changes the ownership recursively. Obviously, substitute your own username for "andre" in the example.
Since I don't use PostGres all the time, I don't want it to be started up automatically. Instead, I set up two aliases in by .bashrc:
alias startpostgres='/usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l /usr/local/pgsql/data/logfile start'
alias stoppostgres='/usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l /usr/local/pgsql/data/logfile stop'
when you start postgres, you'll probably get the dreaded "shared memory error," which looks like this:
FATAL: could not create shared memory segment: Cannot allocate memory
DETAIL: Failed system call was shmget(key=1, size=1081344, 03600).
HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory or swap space. To reduce the request size (currently 1081344 bytes), reduce PostgreSQL's sharedbuffers parameter (currently 50) and/or its maxconnections parameter (currently 10).
some resources will tell you to edit the /etc/rc file, which is incorrect for Leopard. Instead, create /etc/sysctl.conf (if it does not already exist -- mine didn't), and add the following lines:
kern.sysv.shmmax=167772160
kern.sysv.shmmin=1
kern.sysv.shmmni=32
kern.sysv.shmseg=8
kern.sysv.shmall=65536
You'll need to su to root to create /etc/rc
Have fun with Postgres!
Posted in Scout | no comments
Derek
Tracking the results of your blatant self-promotion campaign can be a time-consuming effort. You might be using Google Analytics for web traffic and FeedBurner for blog subscribers. You’re probably checking link referrals. You’re querying the database for usage statistics (user signups, logins, etc.), etc.
Scout is an honest friend that gives it to you straight. Our friendly retriever will track, mash, and graph all of this data in real-time.
For example, below is a graph generated by Scout. It shows the FeedBurner circulation of this blog (in red) vs. unique visits from Google Analytics on our Highgroove homepage (in blue):
It doesn’t look like there’s a huge correlation there. What about unique visitors on PlaceShout (in red) vs. unique visitors on our Highgroove site (in blue)? Data via Google Analytics:
There’s a correlation there. Traffic to PlaceShout appears to drives traffic to Highgroove.
Currently, 3 Scout Plugins exist for grabbing external data:
Seeing this data is extremely useful for answering questions that take quite a bit of work to find out manually (and can’t be updated in real-time):
- How many of our unique visitors create a shoutout on PlaceShout?
- As the number of sites linking to us increases, how does this impact traffic on our site?
- How is traffic impacted when we publish our email newsletter?
The great thing about these reports is they don’t require any updates – Scout continually grabs new data and updates the graph.
Signup for our launch email list
We’re launching Scout this winter – click here to signup for our launch notification. We’ll email that list before the public launch.
Get Immediate Access to Scout
I’d like to create a plugin to report back the number of people linking to a url on del.icio.us, but haven’t had time yet. Want to create this plugin? Shoot me an email at (derek at highgroove dot com) and I’ll give you immediate access.
Past posts on Scout:
Posted in Community, Ruby on Rails | no comments
CBQ
Recently I’ve been reading Obie Fernandez’s book, The Rails Way.
I made an appearance in one of the chapters, but the real credit goes to Obie for pulling together an exhaustive 850-page book.
It’s not for beginners, but if you’ve taken one of my Rails training classes, it’s a great reference book and next step.
Reviews of the Rails Way:
Posted in Scout | no comments
Derek
When something bad happens you want to find out about it as quickly as possible.
You’re probably notified of uncaught exceptions. What about slow web requests, which can be just as annoying to a user?
Find about about slow web requests (and what might be causing them) in near real-time using Scout and the Ruby on Rails Request Monitoring Plugin.
Here’s how it goes down:
1. Install the Scout Client 
watch a video (1 min 39 sec)
2. Install the Plugin watch a video (1 min 55 sec)
...that’s it – you’re no longer a performance slacker.
3. Scout reports back data
Every 10 minutes, Scout collects information:
If you have a slow request, an alert is generated. You can view the offending requests and their request times:
View Web Requests on a Graph
You can easily graph this data as well with Scout’s built-in graph builder:
Compare to other Rails applications
...but that’s not all. You probably have multiple Rails applications. You can compare their performance on a single graph as well:
Look for trends against other data
...we’re not done yet though…how about comparing the average request time vs. the size of the mongrel threads through the Process Usage plugin?
Since it’s easy to plot different data stats on Scout, you can quickly rule out possible reasons for slow performance. That’s half the battle.
Signup for our launch email list
We’re launching Scout this winter. Signup on our homepage, and we’ll give you access to Scout before the public launch.
Andre
I have a geocoding recipe coming out in Pragmatic’s Advanced Rails Recipes . The recipe is titled “Finding Things by Location,” and it offers how-tos and best practices on using my GeoKit plugin to, well, Find Things by Location. The most recent PDF of the beta book includes my recipe along with over 30 other recipes by community leaders like Chris Wanstrath, Dan Benjamin, and Geoffrey Grosenbach, to name just a few.
Posted in Open Source | no comments
James
I had an unusual Christmas Eve. While my wife was entertaining the family, I was in my office programming until nearly midnight. (Have I mentioned how much I adore my wife?!)
What had me so tied down at such an odd hour? One of the open source libraries Highgroove maintains: FasterCSV.
The Ruby core team gave me the nod to replace the standard CSV library with FasterCSV just before the Ruby 1.9.0 release. While a little more warning would have been nice, I was happy to do it.
I think it’s important to discuss why Highgroove, not just allows, but actually prefers that I spend some of my time maintaining our open source libraries, like FasterCSV.
The reason we created the library is nice and simple: we needed it. We import or export spreadsheet data at some point for most of our applications. We work with large data sets and we needed that to be fast. We also frequently use headers in the CSV files to map directly to database fields so a broader feature set to support such things was important to us.
Now we could have built it and just used it internally, but we decided to share. Did we do it because we are just really nice guys? Not really. (We are though, of course!) We did it because we wanted it to be great.
There are many subtle variations and edge cases for CSV. We debugged the library the best we could, but we’re human and we missed things. That meant our library would have trouble with some inputs. We (hopefully) would have noticed those eventually, but we’re just one set of users and by releasing it on the world we multiplied the user base by thousands. Needless to say early users found problems for us much faster than we could have alone. Better than that, they sometimes even fixed them for us! Even when they don’t, they usually send in an example that leads us straight to the problem which is almost as helpful.
That’s not all. Remember all of those features I told you we love? I wish I could brag and tell you I invented all of them, but some of my favorite features in FasterCSV were sent in by the users. I got the ball rolling with what I could see us needing, but the users ran with the ideas and made my library better than I knew it could be.
Highgroove isn’t losing hours on me applying a few patches here and there. They are gaining the help of many extra employees. We love that.
To top it all off, Matz gives us the ultimate Christmas present. By blessing our work, he saves us the trouble of even installing it for new projects. Very soon we will be able to count on our code being available in all modern Ruby installs. That’s just one less thing we need to worry about.
With open source everybody wins.
2 comments
Derek
I’ve been researching ways to consolidate bank account information into a single, automatically updating view. There are a couple of new web-based options – most notably Wesabe and Mint – but those have a consumer focus. I’m more interested in viewing trends in our business – a real-time financial dashboard.
Open Financial Exchange (OFX) Standard
In January of 2007, the Open Financial Exchange standard was announced by Microsoft and Intuit. Many financial institutions provide support for a portion of the standard, namely the portion used by Quicken.
The best list of OFX connection settings is here. A bank information generator is also available that retrieves connection data from the Microsoft Money website.
Retrieving OFX data
Once you obtain the OFX connection details for a financial institution, you can then connect to it to retrieve financial transactions. Getting to this point often isn’t straight forward. For many banks, the online login information isn’t the same as the data needed for the OFX connection. Some banks, like Wells Fargo, charge a monthly fee for OFX access. Finally, chances are your bank’s customer support line won’t know anything about OFX. Your best bet is to ask how to enable Quicken access (which uses the same OFX standard).
There are a couple of libraries available for retrieving OFX data:
- LibOfx is the library used by GnuCash to connect with banks online. Full documentation is available, but I didn’t find any tutorials for getting started.
- OFX for Ruby is similar to LibOfx, but obviously in Ruby. There are a couple of examples included in the library download.
In general, retrieving this data isn’t a straight forward process. OFX connection details are scattered, some banks haven’t made it easy to enable OFX, and there isn’t much information available on building an OFX client. It’s definitely primitive.
Additionally, OFX transactions can be cryptic – a large part of Wesabe’s value is in its ability to provide better transaction formatting.
How are others doing it?
Wesabe and Mint are 2 fairly new web startups that automate the process of collecting account transactions. How they collect the data is very different.
With Wesabe, you record a Firefox browser session, walking through the download process. You can then re-run the session and Wesabe then updates your account data using the script.
Mint provides the most seamless experience. You simply provide your bank details (even PayPal works) and it grabs your transactions. Mint uses “Yodlee” to actually connect with financial institutions. Yodlee has been doing this for a while – they have a service called MoneyCenter that is remarkably similar to Mint – it actually has more features but lacks Mint’s visual appeal. My guess is Yodlee charges a pretty hefty licensing fee.
The Future
I wouldn’t bet on banks making it any easier to automate data retrieval. Erick Schonfeld of Techcrunch recently wrote a post that covered the API landscape very well – which types of companies are most likely to offer APIs and why companies that push APIs are very careful with what they are actually opening up. Opening up a core component of your business can spur some pretty dramatic changes – it will take a younger, upstart bank to spur progress.
While it’s not here yet, I’d pay for a service that:
- Made connecting to financial institutions as easy as Mint (just a user name and password – no OFX connection details needed)
- Scrubbed transactions like Wesabe
- Published an API for retrieving this data that didn’t require a customer account (in other words, the ability to query for bank account transactions from an account not directly setup through the provider)
Posted in Scout | no comments
Derek
I’ve added 2 videos to the Scout homepage – they demonstrate 3 things:
- Installing the Scout client on a remote server is almost too fast
- One-click plugin configuration (I’m installing the Ruby on Rails Request Monitoring Plugin, so if you’re a Rails developer, there’s an added bonus)
- My awkward narration voice (I think my voice actually cracks on one of them…I assure you, I am not 13 years old).
If you’re looking for an easier way to monitor your servers and web apps, check out the videos. There’s a launch notification form on the Scout homepage as well – we’ll email people on this list before publically launching Scout.
no comments
Andre
Much of what Rails provides to get your apps up and running isn’t optimized for performance. It’s crafted to be more efficient for developers, not more efficent at runtime. before_filter callbacks on your RESTful controllers to get the current object? That’s an extra database call. All those nifty plugins you are using to kickstart your app? They probably generate far more SQL (and slower SQL) than if you coded the same functionality ad-hoc. ActiveRecord itself is slow compared to raw SQL and object instantiation.
If your project grows huge—10’s of millions of PV/day huge—you’re gonna have to revisit some of that stuff. Some of it you can compensate for with smart caching techniques and more hardware. And some of it you will have to throw away and rewrite. If you get really huge, you’re going to have pay back some of the technical debt you incurred by choosing a tool like Rails.
Yes, using a tool like Rails incurs some debts. But just like the rest of the world, there are good debts and bad debts. If you’re smart about the kind of debt you take on, you can build far more, and build it faster.
Many of your projects will never reach the level where you need to “pay back” for all that developer productivity you enjoyed on the front end. That means you can try more ideas, and (hopefully) fail fast at the more speculative ones. If some of your projects do need to scale radically (beyond the basics of better caching, more hardware, etc), you have an incredible amount of upfront productivity you can leverage against that work of—say—optimizing some key queries by hand. As long as you go in with eyes wide open and realistic expectations, then I say that’s a smart kind of technical debt.
Posted in PlaceShout | no comments
Derek
One issue we face with PlaceShout, our short-form local reviews site, is that many of the shoutouts may not be relevant to you. If you haven’t been to Nihon in San Francisco, you can’t agree or disagree with Natasha’s shoutout: “Swanky whiskey bar/restaurant. Food was good-small plate Japanese and sushi”.
We recently implemented “Shoutouts You Might Be Interested” functionality to make it more relevant.
Nightly, we scan our database looking for places you’ve been. We find any shoutouts created at those places. These shoutouts are displayed on your personalized homepage, and you can quickly agree, disagree, or pass on each of the shoutouts. There’s a great chance you’ll have an opinion on these places since you’ve already voiced your thoughts on each of them in the past.
Once per week, we email 3 of these shoutouts to you. Directly from the email, you can agree, disagree, or pass:
So far, it’s been very successful. 38% of users have voted on a shoutout from an email. This increased the number of votes on shoutouts by 27% (real-time voting from Scout):
It’s had a great impact for 2 reasons—it keeps our users informed of activity at places they’ve been and it makes browsing and searching PlaceShout better. The most accurate (and least accurate) shoutouts now have significantly more votes.
