a day in the pit my view from inside

21Apr/120

JavaScript, CoffeeScript, and Closures

I've really liked CoffeeScript because it mimics behaviors and syntax I'm used to in Ruby; thus the speed of development for me is skyrocketing as the context switch between Ruby & JS has been minimized. But when I first started mucking around with it a few weeks ago I hit some oddities that I wanted to share with other developers since I've seen the question asked a few times.

CoffeeScript can, if this feature is enabled, wrap your .coffee code inside of an anonymous function that acts as a closure. Rails 3.1 enables this by default. I am also using jQuery, so the $ -> is merely the wrapper that provides the jQuery namespace context.

test.coffee

 
$ ->
  show_dialog = (title,msg) ->
    dialog = $('#dialog')
    dialog.find('h2').text(title)
    dialog.find('p').text(msg)
    dialog.modal('show')
 
  $('#about_us').click ->
    show_dialog('About Us', 'Pinning is fun!')

Becomes

test.compiled.js

(function() {
 
  $(function() {
    show_dialog = function(title, msg) {
      dialog = $('#general_modal');
      dialog.find('h3').text(title);
      dialog.find('p').text(msg);
      return dialog.modal('show');
    };
    return $('#about_d').click(function() {
      return show_dialog('About Us', 'Pinning is fun!');
    });
  });
 
}).call(this);

Uh oh. This creates a problem: if you want your view to call one of the functions you have defined in a CoffeeScript file you're SOL unless you make those functions globally available (I don't encourage cluttering the global namespace). Causing further headache, when you realize your functions are not being called and try to debug things you find yourself looking at reference/undefined method errors. Solution: throw your functions (JS or CS) in a namespace and have globally accessibly, albeit scoped appropriately, JavaScript. And, for added benefit, have a cleaner code base :)

(Displaying actual code to give you more context of how I use this solution)

welcome.js.coffee

 
window.Pinned ||= {}
 
$ ->
  Pinned.handle_invite_request = (json) ->
    console.log(json)
    console.log(json.status)
    if json.status == 'success'
      $('#invite_success').html("We'll e-mail <b>"+json.email+"</b> when we're ready!").fadeIn('fast')
      $('#invite_request').slideUp('fast')
    else
      $('#invite_error').html(json.message).show()
 
  Pinned.request_invite = () ->
    $.get("/welcome/request_invite",{'email':$('#invite_email').val()}, Pinned.handle_invite_request, "json")
 
  genModal = $('#general_modal')
 
  Pinned.show_dialog = (title,msg) ->
    genModal.find('h3').text(title)
    genModal.find('p').text(msg)
    genModal.modal('show')
 
  $('#about_d').click -> 
    Pinned.show_dialog('GetPinned',"We believe there is a better way to encourage sharing, collaboration, and love for all things artsy and digital. Our passion is to help you share your passion.")

Becomes

welcome.js

(function() {
 
  window.Pinned || (window.Pinned = {});
 
  $(function() {
    var genModal;
    Pinned.handle_invite_request = function(json) {
      console.log(json);
      console.log(json.status);
      if (json.status === 'success') {
        $('#invite_success').html("We'll e-mail <b>" + json.email + "</b> when we're ready!").fadeIn('fast');
        return $('#invite_request').slideUp('fast');
      } else {
        return $('#invite_error').html(json.message).show();
      }
    };
    Pinned.request_invite = function() {
      return $.get("/welcome/request_invite", {
        'email': $('#invite_email').val()
      }, Pinned.handle_invite_request, "json");
    };
    genModal = $('#general_modal');
    Pinned.show_dialog = function(title, msg) {
      genModal.find('h3').text(title);
      genModal.find('p').text(msg);
      return genModal.modal('show');
    };
    return $('#about_d').click(function() {
      return Pinned.show_dialog('GetPinned', "We believe there is a better way to encourage sharing, collaboration, and love for all things artsy and digital. Our passion is to help you share your passion.");
    });
  });
 
}).call(this);

You can now reference your functions that have been defined anywhere in your code!

12Apr/120

Two ways to Saving E-mail Attachments (AppleScript & Ruby+EWS)

I had this fun project of storing & processing some financial analysis reports. The goal was to receive periodic (bi-weekly) reports that were sent by an automated reporting software. The catch was that these reports were only deliverable via e-mail and the e-mail address they were being delivered to was a Microsoft Exchange account with no POP/SMTP access. Since I had recently been mucking with my Mac Mail.app rules I figured I'd write a little AppleScipt... Check it out:

(My dev machines already talk to each other over authorized SSH keys so this became a snap)

on perform_mail_action(ruleData)
	tell application "Mail"
		set theMessages to |SelectedMessages| of ruleData
		repeat with themessage in theMessages
			set theAttachments to mail attachments of themessage
 
			try
				repeat with anAttachment in theAttachments
					repeat 1 times
						if name of anAttachment does not end with ".htm" then exit repeat
						set {seconds:s} to (current date)
 
						set thePath to "/tmp/reports/" & s & (random number from 10000 to 900000) & name of anAttachment
						save anAttachment in thePath
						set cmd to "scp " & thePath & " ov2:/opt/svn/optisol/src/ruby/transit/reports"
						do shell script cmd
					end repeat
				end repeat
			end try
		end repeat
	end tell
end perform_mail_action

So my fun with AppleScript was short-lived. To have a more reliable solution I switched over to the the pretty decent SOAP API for Microsoft Exchange called Microsoft Exchange Web Services Managed API (or just EWS). And luckily a neat Ruby Gem also exists called Viewpoint that makes life a breeze. In just a few lines of code, here is my script that pulls down a search for the last 7 days of e-mails that have attachments and are sent to a special mailing address:

require 'viewpoint'
require 'kconv' # If on Ruby 1.9
 
Viewpoint::EWS::EWS.endpoint = EWS_URL #... 'https://yahoo.com/EWS/Exchange.asmx'
Viewpoint::EWS::EWS.set_auth EWS_USERNAME,EWS_PASSWORD
 
inbox = Viewpoint::EWS::Folder.get_folder_by_name('inbox')
 
restrict = { 
  :restriction => { 
    :and => [
      :is_greater_than => [ {:field_uRI => {:field_uRI=>'item:DateTimeSent'}}, {:field_uRI_or_constant => {:constant => {:value=>DateTime.now-7}}} ],
      :is_equal_to => [ {:field_uRI => {:field_uRI=>'item:HasAttachments'}}, {:field_uRI_or_constant => {:constant => {:value=>true}}} ],
      :contains => [ {:field_uRI => {:field_uRI=>'message:ToRecipients'}}, {:constant => {:value=>'your@email.com'}} ]
    ]
  } 
}
 
search_results = inbox.find_items(restrict)
puts "#{search_results.size} items to inspect/process"
 
search_results.each do |email|  
  # Attachments are Base64 encoded -- let's unpack and grab the result
  attachment_content = email.attachments.first.unpack('m').first
 
  puts attachment_content
 
  # Viewpoint also gives you a handle save method
  # email.attachments.first.save_file
end

Here are some helpful pages worth referencing

Filed under: Code, Mac OSX No Comments
21Mar/120

Self Signed SSL Certificate, nginx, and RightScale

Here's my go-to article on creating self-signed SSL certificates, using them in NginX, and enabling secure access to a RightScale VM. You'll need OpenSSL to follow this very brief tutorial.

Step

Ensure /etc/ssl/ exists. It may already be present; if not create it. And create a local working directory ~/ssl_fun and cd into it.

mkdir /etc/ssl/
mkdir ~/ssl_fun
cd ~/ssl_fun

Create a Private Key

You'll need to enter a pass phrase. Pick something. We'll remove it later for nginx.

openssl genrsa -des3 -out secure_cert.key 1024

Create a CSR (Certificate Signing Request)

You'll be asked a series of questions. Enter what you want. However, for the common name you should enter the domain (or subdomain.domain) that you want to use this certificate for. You can use this certificate with _any_ domain, however you will see a few more errors if the Common Name does not match the domain you're using.

openssl req -new -key secure_cert.key -out secure_csr.csr

Remove Pass Phrase for Nginx

I lied. This isn't for Nginx. But it allows Nginx to work with this certificate without needing the password for such cases like server reboots. Enter the pass phrase you picked in an earlier step.

openssl rsa -in secure_cert.key -out secure_cert_nginx.key

Get the CRT

Last step for the generation process is here. Let's create the CRT and then we can move our files around to work with Nginx.

openssl x509 -req -days 365 -in secure_csr.csr -signkey secure_cert_nginx.key -out secure_cert_nginx.crt

Copy & Go

The CRT file goes to /etc/ssl/certs and the Key file goes to /etc/ssl/private...

sudo cp secure_cert_nginx.crt /etc/ssl/certs/
sudo cp secure_cert_nginx.key /etc/ssl/private/

If the folders don't exist then create them!

Cleanup & Update Nginx

You can now remove ~/ssl_fun safely. Next, fix up your nginx.conf file. I made the server block for port 80 forward to port 443 and then moved my port 80 settings to the new port 443 server block:

    server {
        listen 80;
        server_name findbyimage.com www.findbyimage.com;
	rewrite ^(.*) https://$server_name$1 permanent;   
    }
    server {
        listen 443;
        server_name findbyimage.com www.findbyimage.com;
        client_max_body_size 50M;
        error_log /home/logs/okc_browser_errors.log;
        root /home/rails/okc_browser/web/public;
        passenger_enabled on;
        location ~* \.(ico|css|js|gif|jp?g|png)(\?[0-9]+)?$ {
            expires max;
            access_log off;
            break;
        }
	ssl on;
	ssl_certificate /etc/ssl/certs/secure_cert_nginx.crt;
	ssl_certificate_key /etc/ssl/private/secure_cert_nginx.key;    
   }

A quick reload of the nginx binary and we were good to go (this step may vary depending on how you install Nginx):

/usr/local/nginx/sbin/nginx -s reload

RightScale specific step

I did not have port 443 in my security group so I had to add it. Check out Add IP Address Based Permissions on the RightScale Support Site.

NJOI

Filed under: Learning, Software No Comments
25Feb/122

Saturday Scrape — Surf Reports

It's Saturday. The surf is bad today which is why I decided to write a surfline.com scraper. It's a freestyle post written while I code so don't sweat the small stuff. This is entirely a quick, dirty solution to getting some data to play with.

Tools used during this session:

  1. Ruby 1.9.2
  2. Nokogiri (HTML Parsing)
  3. Typhoeus (HTTP Library for Fetching HTML)

First... Why write a scraper? No API exists for SurfLine.com and I want data.

SurfLine.com offers two ways of accessing data:

  1. Consume the web page HTML
  2. Consume the widget HTML

Each present their own problems. As I go through this short tutorial I'll show you when things change and why knowing how to pivot is important.

The data that I want (right now) is the height of the waves for a surfing spot I frequent. The URL is http://www.surfline.com/surf-report/38th-ave-central-california_4191/ Hey, cool, turns out the height is wrapped in a nice identifiable DOM element:

<p id="text-surfheight">1-2 ft</p>

So a quick XPath selection on the Nokogiri::HTML document gets what we want...

elem = Nokogiri::HTML(page).search("//p[@id = 'text-surfheight']")

elem now contains an array of the elements we found from our search. Let's pull the first one out and grab the inner_text

elem.first.inner_text

We're done right? Unfortunately, surf reports are user reported and not always in the format we'd expect. I quickly discovered some pages don't contain a text-surfheight id, but instead a short sentence describing the height:

<p class="text-data bottom-space-10">Inconsistent occ. 2 ft. </p>

That's frustrating since now our code can't simply look for the same element every time. So we improvise. Instead of spending time figuring out how to triangulate what I want out of this big page I start to look and see if there is a widget or API that could give me the surf report; there is a widget service. It makes a JavaScript call to load an HTML iFrame up. Great. So I jump right in and check out the new HTML page I'm looking at. The good thing about this widget is it's only the surf report and not a bunch of web site features, videos, and links that I don't need to look at. And, most importantly, the widget displays something about the wave height _always_. Unfortunately, the widget HTML is disgustingly ugly and has no apparent patterns. Sometimes the surf report height is contained in a span element and other times it's thrown into a div; neither have ids. Iterating over a few different surf report pages I find that the widget does have one pattern: CSS Styling. (I know. Yucky)

But, the nice thing about extensive hardcoded styling in HTML is that it can actually serves as uniquely identifiable keys when looking at a small amount of html (like a widget!). So we can write an XPath search:

# Helper method to take a Nokogiri search and return nil
# or the value of a non-empty element
def inner_text nokogiri_search
  nokogiri_search.first.inner_text rescue nil # exist or set to nil
end
 
# spot_id comes out of a hash. Check out the full code linked @ the
# bottom of this page to see more
n = Nokogiri::HTML(
  grab_page(
    "http://www.surfline.com/widgets2/widget_camera_mods.cfm?id=#{spot_id}&amp;mdl=0111&amp;ftr=&amp;units=e&amp;lan=en"
  )
)
 
height = inner_text(n.xpath("//span[@style='font-size:21px;font-weight:bold']")) ||
  inner_text(n.xpath("//div[@style='font-size:12px;padding-left:10px;margin-bottom:7px;']")) ||
  "Report Not Available"

If the first clause passes than we have a wave height. If the second conditional passes we have a short sentence describing the surf conditions. If we can't find anything we just default to "Report Not Available"

Okay, so it is not pretty but we've now got a decent way to identify wave heights from surf reports on Surfline.com's Widgets which I've tested across 10 surf spots and seems to work OK for this initial prototype.

What's next? Adding in the Tides table on the widget. It's also a fun trickster since you have to look for the test "TIDE," take first search result, and grab the parent element:

tides = n.xpath("//div//small[contains(text(),'TIDES')]").first.parent

Which gives us:

"\nTIDES:\n\n \n \n \n \n 02/24\u00A0\u00A0\u00A005:48AM\u00A0\u00A0\u00A01.23ft.\u00A0\u00A0\u00A0LOW\n \n \n \n \n 02/24\u00A0\u00A0\u00A011:46AM\u00A0\u00A0\u00A04.45ft.\u00A0\u00A0\u00A0HIGH\n \n \n \n \n 02/24\u00A0\u00A0\u00A005:49PM\u00A0\u00A0\u00A01.07ft.\u00A0\u00A0\u00A0LOW\n \n \n \n \n 02/25\u00A0\u00A0\u00A012:07AM\u00A0\u00A0\u00A04.97ft.\u00A0\u00A0\u00A0HIGH\n"

That looks ugly. Why is there Unicode in there? Let's pull out just what we want...

prettier_tides = tides.text.gsub("\u00A0\u00A0\u00A0"," ").scan(/\d(.*?)\n/)
# => ["02/24 05:48AM 1.23ft. LOW", "02/24 11:46AM 4.45ft. HIGH", "02/24 05:49PM 1.07ft. LOW", "02/25 12:07AM 4.97ft. HIGH"]

What you do with this data is now up to you. I store it in a SQLite database and run the script every hour or so to get updates form 8 am to 2 pm PST.

ShouldISurf GitHub and for the scraping code you should look at lib/grab_reports.rb


As of 4/12/2012 this code has been running daily for almost 2 months and serving up surf tides on shouldisurf.com. Let me go knock on wood. Okay, back. The code base is small and effective. I'm glad I didn't invest any time in making a robust solution now!

26Jan/120

HelloBirthday Grows Up, Goes Private

It must have been two years ago that I missed my friends birthday and she was super upset with me. Naturally, I devoted 48 hours to building an app that would never let me forget a birthday. (I think this may have pissed her off even more...)

Fast forward... I've decided to change HelloBirthday so that as of January 25th, 2012 new users will only see a forecast of birthdays and wishing will not occur. This only affects new users; current users you're OK.

HelloBirthday still has the capabilities to automate wishing and I'm letting friends, family, and people who know me to continue to use it. If you want to use HelloBirthday please add it and then e-mail me (mike@thinkeffect.com).

20Jan/120

I hope you’re surfing

The Rush

I've been thinking about how thrilling life can be. One minute you're in a comfort zone without realizing it and then in a flash you're paddling your heart out head-on for a large body of water that's trying to crash down on you. And as you paddle into this wall of a wave your instinctual response, to the sheer amount of fear amongst other things, is to paddle, kick, and dive into the belly of the beast. Just barely scraping past the wave you and your board flop down onto the hard surface water only to keep paddling because you know there is another wave coming; the set had just begun.

A freezing chill runs down your spine but you realize it's just some of the water from that last wipe out. Fuck, it's freezing cold.

But you continue because the feeling of this challenge has never felt better. It's why you put yourself in situations you're never comfortable in, and why you know the devil in comfort intimately. These waves don't care about anything except providing you challenge after challenge with no end in site.

Ride a Wave

Surfing is a lot like entrepreneurship. It's fueled primarily by you and your own will, and encouraged by that group of people in the water with you. It's still somewhat of a dog-eat-dog world but that bond between soloists is stronger than one might think. It's incredibly rewarding. It takes an immense amount of time to do it well but you can start and feel empowered immediately.

Good luck. Have fun. And remember to share the waves.

2Nov/110

What’s New in Rails 3 & What I’m Excited About

Rails 3.1.0 was released on 8/31/2011, and as such marks a great day for the Rails community. For a while Rails felt stagnant (think 2.3.11 to 3.0.1RC) and so this is something I've been looking forward to. As I've been using Rails 3 for over a year now, and I've been following along in the change sets, I wanted to point out some of the features I think are really going to be game changers.

Sprockets, and the Asset Pipeline

Previously done through third party libraries, the Asset Pipeline is a built-in framework for managing your assets and writing these assets in other, some say more friendly languages, like CoffeeScript for JavaScript and Sass for CSS  style sheets. It's a very large change to Rails because it introduces a new mix of options for how you can write your JS/CSS and it moves the serving of these components to the Rack middle-ware. Your asset resources now can be pre-processed, minified, and compressed in one swoop. This process is done by Sprockets. I won't go into any further detail but you should know this is worth reading up on, so go check out the Asset Pipeline introduction by the RoR team. (You can disable this feature if you don't want to use it. So don't freak out!)

Streaming

Although it requires Ruby 1.9x to run, HTTP streaming has finally been added. Part of the confusion I often hear about Rails is why this feature was not there from day one. To be honest, I'm not sure but my hunch is that it didn't make sense in a prototyping stage to have to stream content. Further, it's very very error prone compared to building your response and then shipping it over (i.e. if computational errors occur mid stream you're dead in the water and the page will never finish loading). Further, Ajax helped mitigate this need by loading a light HTML shell and then using asynchronous calls to fetch your users' data. At any rate, I'm very excited for this feature because the last two years of PHP coding has had me used to buffering output and I really do see the value in being able to use streaming to show progress without making lots of asynchronous calls.

JSON

ActiveResource now defaults responses to JSON, as opposed to XML.

jQuery

Is now the default JavaScript library bundled with Rails 3. Further, RJS has been factored out as a gem.

Basic Authentication

Rails 3 comes with a quick and easy way of doing Basic Authentication (Username/Password) in your Controllers. Read up on Base.http_basic_authenticate_with - Check out the example here

Pluralize Names for Models

Yup! You can now set, on specific models, whether you want them pluralized or not. From within your controller class you'd set: self.pluralize_table_names = false

BCrypt Passwords

You now have a model attribute has_secure_password that will take care of password hashing/encryption.

3Oct/112

State of the Union

It's been some time since my last post. Yeah, I know, that may be a good thing for some of you and bad for others! Either way, I'm back in school temporarily to finish up my last class to get my undergraduate degree. What this really means is that I'm now throwing a new ball into my juggling routine. Will I survive? Eh, most likely. But for those who have been bitten by curiosity here is what I'm up to now-a-days:

I'm still working full-time for Yahoo! as a software developer/advertising analyst. I'll continue my night-time hacks; but these will slow down. I'm completely immersed in my computational models & theory course at UC Santa Cruz. I recently picked up an Ipad2 to begin developing on the mobile platform and have begun working on a game. And last but not least I continue to eat, sleep, and surf.

So there you have it. A breakdown of all that is important to me right now and why this blog may just experience a pause for silence. Who knows, I just might have some interesting things to say when I resurface in a few weeks. Surely, you can expect some exciting news towards December :) You should follow my twitter account @smasher5 for snippets of interesting (and uninteresting) things.

Katt Williams said life is too fucking short. Yeah Mr. Williams I agree with you. Therefor I say to all of you reading this post, "go outside." (And make your paper booboo)

Filed under: Information 2 Comments
30Aug/110

Resque me with a custom Redis host

If you're working outside of Ruby on Rails and want to have Redis hosted non-locally than this is the command you're looking for:

Resque.redis = '<non_localhost_address>:<port>'

To paint the full picture: You've got Redis running on Machine1.Hostname and your Resque workers on Machine2.Hostname and you need those two wired up. In your Rake file for Resque workers you should add the line of code from above.

One little caveat, if you're told Resque is an uninitialized constant than you just simply need to get Resque into your environment. Here's an example IRB session where I'm using Bundler/Gemfile:

ree-1.8.7-2011.03 :001 > require 'rubygems'
 => false 
ree-1.8.7-2011.03 :002 > require 'bundler/setup'
 => true 
ree-1.8.7-2011.03 :003 > Bundler.require
=> [<Bundler::Dependency type=:runtime name="SystemTimer" requirements=">= 0">, <Bundler::Dependency type=:runtime name="rake" requirements=">= 0">, <Bundler::Dependency type=:runtime name="redis" requirements=">= 0">, <Bundler::Dependency type=:runtime name="redis-namespace" requirements=">= 0">, <Bundler::Dependency type=:runtime name="yajl-ruby" requirements=">= 0">, <Bundler::Dependency type=:runtime name="resque" requirements=">= 0">, <Bundler::Dependency type=:runtime name="ruby-prof" requirements=">= 0">, <Bundler::Dependency type=:runtime name="json" requirements=">= 0">, <Bundler::Dependency type=:runtime name="nokogiri" requirements=">= 0">, <Bundler::Dependency type=:runtime name="fastercsv" requirements=">= 0">, <Bundler::Dependency type=:runtime name="yieldmanager" requirements=">= 0">]

Cool, now we've got our environment and can do what we want. It's not necessary to use a Gemfile, you can manually install your gems and then require them in your Ruby file. But since I have multiple files doing many different things it makes sense to have one common section for inclusion of my packages. I also encourage creating a ruby include file with your Redis setups; this will prevent you from having that Resque.redis scattered in all of your files that access Redis.

Hope this saves someone a minute. More information is in Defunkt's Resque package on Github.

Filed under: Code, Redis, Resque, Ruby No Comments
20Aug/111

Visualizing Subversion’s Commit History

Yeah, it's Friday! Woohoo. Yup, that's right people, it's time to kick off your shoes and code...

I have not posted in like two weeks. Lemme tell ya, One of those weeks I spent in Vegas on a much needed vacation without a laptop. It was pretty spectacular. This week has just flown by. But today I felt like doing something fun; I felt like being a movie producer, director, composer, etc. However, I don't really like to get off the couch just to film a movie when I have a at least 25 storylines wandering around my computer.

Without further pause here is my story of the main Subversion repository I contribute to at work:
[ By the way, dots (or nodes) are directories or files. As the two people-icons run around the screen they are making subversion commits. Explosions typically occur when a big check-in happens. ]

Enjoy? Let's make you a video. Visualizing Subversion commit activity is crazy simple. Dayum! I'm running Mac OS X 10.6.8 so all of following was done on that environment.

You'll want to get ffmpeg and some codecs. I use Darwin Ports to manage packages like these so if you're like me here's the quick fix

sudo port install ffmpeg +gpl +lame +x264 +xvid

Next, we're going to need Gource, and ffmpeg.

Download gource. unzip it and head into the directory. once in the gource directory go ahead and

sudo ./configure
sudo make; sudo make install
gource --help

Alright we're pretty much ready to go. Head over to one of your subversion root directories. Add this file to your directory: summerfun.conf
and in it put

[gource]
bloom-intensity=0.25
colour-images=true
hide=filenames,dirnames
path=my-project-log.xml
seconds-per-day=0.1

You'll see the my-project-log.xml is nowhere to be found. Let's create it:

svn log -r 1:HEAD --xml --verbose --quiet > my-project-log.xml

That will make the video play a little faster. You can remove or muck with all of the settings; just read the README for available options. Now here is the command I executed:

gource --load-config summerfun.conf -1280x720 -o - | ffmpeg -y -b 10000K -r 60 -f image2pipe -vcodec ppm -i - -vcodec libx264 -vpre default -threads 0 -bf 0 gource.x264.mp4

This command launches a video; watch it & interact with it. Give it a try. All interactions get recorded in the video output. So zoom as you wish, to entertain your watchers. Let me know how it goes and post your videos.

Reference: