Setting up a daily task on Heroku

When I set out to create the Skills App site, one of the main things I wanted to figure out was how to scrape the data on a daily basis and automatically insert into the database, preferably at the same time each day. I started out with some quick googling while I was developing the app, but nothing obvious stood out. There were a couple of ways that seemed to be repeated on how to get data into a rails app. One was having a rake task, and the other one was via a seed file. I wasn't sure what to do so I tabled it, and decided that it would be one of the first things that I figure out after the initial deploy.

After the deploy, my good friend Scott Parker, who makes awesome dragon games, gave me some feedback on the site after reading my initial post.  He suggested that I take a look into using rake and the scheduler add-on for Heroku. I took a look at rake and played around with it a bit. I also found a useful railcast about it which helped me figure out what I needed to do.

So I took the majority of my scraper code and put it into the rake task and tested it locally to make sure it was working, then I set up the scheduler to run around midday, uploaded all of the data I have been collecting every day and anxiously waited to see if it would automatically add today's data.

Success!  Data from today!

Enjoy the site and please feel free to reach out to me with feedback, feature requests, or anything else.  For those that have given me feedback, I really appreciate it and hopefully I will implement your features soon!

Deploying to Heroku

When I deployed my Skills Compiler rails app the other day, I ran into a few issues deploying to Heroku that were worth mentioning.  

1. Deploying from a folder that is not the root.
   
   As stated on the Heroku website:

         Heroku apps expect the app directory structure at the root of the repository. If your app is inside a subdirectory in your repository, it won’t run when pushed to Heroku.

   I knew there had to be a way around this. As someone that has created lots of project structures in their career, I typically don’t put the main app in the top folder. I have found that you can paint yourself in a corner by doing that. When I create a project, I want to include everything that is necessary for it, which could mean other apps, documents, or whatever.
   
   I ended up finding this site: https://coderwall.com/p/ssxp5q that had the same issue and solved it by pushing a subtree. It is as simple as this command:
   
   
   git subtree push --prefix subfolder heroku master


2. Make sure your git ignores are correct.
   
   When I pushed to Heroku using the command above, nothing ended up working.  The console logs weren't much help (just kept giving me generic errors), and when I used the console to list the files, everything appeared to be copied over correctly.  After some digging I realized that none of the rails commands were working on Heroku.  I eventually found an error that pointed me to an issue with something missing from the bin folder.  I found that because I was using the default .gitignore file from GitHub with a few altercations, that the /bin folder was globally ignored due to .Net & (I believe) Java.  After a few modifications to the .gitignore file I finally got my site up and running on Heroku.

Skills Application

I have been working on this application for a few weeks now, and I am finally ready to release version 1.0 of it.  The site, Skills Compiler , (I suck at naming) was inspired by a few things, but these in particular. I wanted to teach myself a new language, I wanted to do some more general web development, and finally I was inspired a bit by the Passionate Programmer: Creating a Remarkable Career.  If you haven't read the book, I definitely recommend it.  In the book, one of the points it makes is keeping up with latest trends making sure your skills are up to date, and it kind of inspired me to think about ways of finding the trends in the development community, which is something I personally wanted to improve on.  

All of this combined with my curiosity started me down the path of creating this app.  The app displays data that I have scraped from the Careers 2.0  in a few different ways.  The app displays the totals per day, as well as the overall totals for a given tag.  A tag is counted when it is listed for a given position.  For example, this position by Sonoma Partners for a Salesforce Developer would add 1 to the daily count for Java, visualforce, apex-code, soql, and JavaScript for each day the job is listed.  

Obviously this is just a small snapshot of the jobs that are out there because it currently only looks at one site, however there are still some interesting things that you can gather from a month and a half worth of data.