Linq 2 You

Paperclip, Heroku, and AWS-S3

2011-12-29T11:20:00.001-08:00

I recently had a requirement to build an “avatar for users” feature. The application is a Rails 3 app running on Heroku. Easy enough. I chose to use the Paperclip gem provided by ThoughtBot. Paperclip’s default is to save uploaded files to the filesystem. The catch, Heroku doesn’t let you write to the filesystem, and I couldn’t use Gravatar. Well, there are a couple of ways to work around this Heroku limitation. One is to save the file to the database as a blob. There are pros and cons to this approach. The other approach, and the one I chose, was to use Amazon’s AWS-S3 service.

Setting up Paperclip

Setting up paperclip is pretty easy. There are various ways to do it, but I’m going to discuss how I did it.

First I added the paperclip and aws-s3 gems to my project’s Gemfile.

Then run bundle to install the gems.

Paperclip stores metadata about the uploaded files into the database, and therefore needs fields added to a table to do so. I added those fields right along side the fields in the user table. I didn’t have the need to track past history of upload avatar images. So this was an acceptable approach. Installing the paperclip gem gave us a generator to help us create the database fields we need. Run the following command in a terminal window.

$ rails g paperclip user avatar

Lets quickly break this command down.

rails g is a short cut for rails generate. You should know this.
paperclip is the name of the generator we want to run
user is the table we are going to add the fields to. For me this table already existed.
avatar is the prefix name we want to use for the database fields

This will create a migration file that adds the necessary fields to the user table, or what ever table name you supplied the generator.

Now you can run this migration.

$ rake db:migrate

The last thing needed to set up the paperclip gem is to add a little bit of code to the model that is going to be responsible for the attachment. In my case, this is the user model. The little bit of code to add is the has_attachment_file method. As you can see from the gist below the code is pretty simple, and maybe even a little self explanatory. In traditional Ruby fashion, this gives you a bunch of extra functionality that you can take advantage of in your views. Not going to cover that here though.

Take note of a few important items in the code.

storage tells paperclip where to save the uploaded file. Default is the file system. Here I’ve specified it as s3.
bucket is the name of the location in the AWS-S3 service that the file will be saved to. Think of it as a folder in the cloud.
s3_credentials are used to authenticate to the AWS services. There are a couple of ways to set the credentials. I used a hash and set the access key and secret key individually for reasons that we’ll cover in a bit. You can also use a Yaml file here as well. See paperclip documentation for that.

And don’t forget to whitelist the avatar property for mass assignment by adding it to the attr_accessible list. See the gist above.

Create an AWS Account

You are going to obviously need an AWS account. I’m not going to go through all the steps to create an AWS account, but if you made it this far you should be pretty committed to going through the process. It’s a pretty cheap service to use, and you only pay for what you use. New users get the first year of S3 use for free. To date, I’ve spent $0.14 for it. Not a type-o, yes 14 cents. Here is the link to the AWS-S3 service. When you sign up for an AWS account you will be assigned an access key and a secret access key. I try and keep both private. The more important one is the secret access key. You can change this if needed, or even add more keys if that is a requirement. You can find your keys by going into the Security Credentials screen.

Once you find them, keep them handy, you’re going to need them in a bit.

Shell scripts – Local and Heroku

This is where things got a little different for me. Since this is an open source project on GitHub, I didn’t want to have any reference to my AWS keys any where in my source code. Hence the reason for using ENV variables that we saw earlier.

Local Development

In Rails applications you can set ENV variables at start up and have them exist for the duration of the application’s life. As an example.

$ S3_KEY=123456789ABCDEF S3_SECRET=x+yabld235lll654 rails s

Now two environment variables exist with the names S3_KEY and S3_SECRET. If you review the user model gist above you’ll see how you access these two values. You probably see right away typing this at the command line is error prone and very tedious every time you want to start the server. Sure there’s the bash history. But, I chose to make a simple shell script to start the server and set these variables.

Important: First add the *.sh file extension to your .gitignore file. This will ensure you don’t accidently check your AWS keys in and upload them to GitHub.

Next in the root of your application add the following command line, replacing the appropriate values with your AWS keys, to a file with the .sh file extension. I used run.sh.

S3_KEY=<your access key> S3_SECRET=<your secret access key> rails s

Save and then set this file as executable.

$ chmod +x run.sh

Test it. If all works, your application will start up as usual. I like this cause it doesn’t really interrupt my workflow.

$ ./run.sh

Heroku Production

The process is very similar as it was for the local development section above. For brevity I’m going to list the steps out instead of going into detail.

Create a .sh file in your application root. I used deploy_heroku.sh.

Add the following lines to that file.

git push heroku master
heroku rake db:migrate
heroku config:add S3_KEY=<your access key>
heroku config:add S3_SECRET=<your secret access key>

Make the file executable

$ chmod +x deploy_heroku.sh

Test it.

$ ./deploy_heroku.sh && heroku open

I loaded this script up with a few more things. I got tired of typing each step out every time I wanted to deploy to Heroku. You are free to change these steps any way you want. The important two pieces are where the config settings are getting set.

Conclusion

I hope you found this helpful and that it all makes sense. Please leave a comment below if you like this, have a suggestion to improvement, or if you have any questions.

References

http://devcenter.heroku.com/articles/s3
http://www.labnol.org/internet/tools/amazon-s3-buckets-tutorial/3890/
http://www.railstoolkit.com/posts/fancyupload-amazon-s3-uploader-with-paperclip

A new chapter

2011-08-27T06:09:00.001-07:00

Next week Wednesday Aug 31st I start a new chapter in my computer career. I’m going back to school. Well, kind of. I’m taking a Ruby on Rails class. It’s a full semester 3 credit course, but I’m only taking it as a continuing education program. The class itself is an elective for the college’s computer science program. In order for me to take it I had to get an instructor authorization for enrollment. But no biggy there. I’m very excited for this. The Ruby on Rails community has always intrigued me. And there are .NET and Java developers moving to Ruby on Rails in droves. Over the past year or so I’ve dabbled with Ruby on Rails from time to time. But I never felt confident enough with it to take on a big project. It is my hope that this class will get me to a more comfortable level.

Last weekend I attended the first ever Madison Ruby conference. It was AWESOME!!! Very enlightening being around that crowd. I met some great people and discovered some new opportunities that I can get involved with. For example the Mad-Railers group that meets here in town. The Mad-Railers are a Ruby on Rails user group. They meet the last Monday of each month. Very excited for that as well.

It is my intention to do some more blogging over the next few months. Mostly with my experiences as a .NET developer moving towards Ruby on Rails. There is a mind shift that has to take place.

Performance issues with Visual Studio 2010

2011-07-01T05:31:00.001-07:00

I’ve been noticing some performance issues with Visual Studio 2010. Specifically when I type. The editor wasn’t able to keep up. And I’m not a super fast typer, 55 wpm. I have a few plugins installed, mainly ones you can get via the Extension Manager along with DevExpress’ Code Rush Pro. I thought maybe it was the accumulation of all the plugins trying to do their thing behind the scenes. I’ve tried disabling or uninstalling plugins, but that didn’t really help. Plus once you get use to using the added features that the plugins provide, you start to feel crippled after they are gone. So I searched around a bit, and found that the culprit was, “My video card”???? Turns out its an easy fix. Tools –> Options. I unchecked the three check boxes from below. Restarted Visual Studio, and that fixed it.

Caliburn.Micro Series

2011-06-10T08:08:00.001-07:00

I’m working on a MVC3 personal project that exposes some services. The reason for the services is because I see a need for a desktop (WPF) client as well. A few months ago I was introduced to the Caliburn.Micro project. I don’t know much about it, but from that discussion I got pretty interested in it. As I was Googling around for some good documentation, tutorials, or code examples I ran into a Soup to Nuts series on Devlicio.us. Below is a TOC of the series. I put this together because over the next few months I’ll be using this as my reading guide. I also noticed there is no part 4. Not sure whats up with that. I’ll update this post if I find out more.

CQRS Presentation video

2011-06-08T13:25:00.000-07:00

I have completed the editing and uploading of my CQRS presentation from 6-1-2011. It took me some time to edit the video, and get it uploaded, but I think the result is worth the effort. I have also included the Power Point slides for you to download. I hope you all like it, and any comments are welcome. As I watched the video back, I noticed a few places where I miss spoke. I'm going to do some follow up articles to clarify the instances where I miss spoke, and also to answer some of the questions that came out of the meeting. I believe there were some good questions that came out. I've been thinking about them and I've come up with some answers to a few of them. So be on the look out for those follow up posts.

Thanks again to all that attended.

Free .NET Libraries

2011-04-26T08:27:00.000-07:00

There is a great list of free .net libraries posted at Qink that you can use in your applications. Check it out. The list is continually being updated from the comments.

Current Packages for the Nu Package Management

2010-08-09T07:54:00.001-07:00

I’ve been doing Rub on Rails now for about 3 months, and I’m really getting into it. So many nice libraries and conventions to help get applications out the door faster. Unfortunately my RoR work doesn’t pay the bills yet. By day I’m a .NET developer. Recently I came across a blog post written by Ayende Rahien that mentions the use of a new package management utility for .NET developers. For years Ruby on Rails developers have enjoyed the luxury of installing libraries using the gem utility. Its really handy to have at your disposal. Read up on it. There is now a similar package management utility available for .NET developers called Nu, short for Nubular. If you read Ayende Rahien’s blog you’ll see how it works.

Current Nu packages are available here.

I hope this project catches on and the list continues to grow. This is going to make things really handy and really quick to get the latest versions of the most common libraries that I use for development. Best of all now I don’t have to do the package management myself on some file share of mine.

Git back here

2010-08-04T08:20:00.001-07:00

If you used the git rm command, ignored the warning, you will notice that the file doesn't get removed from your repository, it gets removed from your file system. There is no trash bin with git either. You might be in luck if you did a git add first, then issued the git rm command. Your file will have been hashed and added to the git staging area. You can retrieve that file by issuing the following command:

git fsck --lost-found

This will pull all the files out of the staging area and drop them in the folder .git/lost-found/other. One catch though. The file isn't named as you would think, or hope. The file is named with its hash value. This is ok, you haven't lost the contents of the file, just the file name. If you poke around in the files a bit, you'll find what you are looking for. Its not perfect, but the content is there. And yes, I did do this.

Uploading files with ASP.NET MVC

2010-06-23T09:46:00.001-07:00

Dino Esposito has posted a great article on DotNetSlackers.com on how to Upload Files using ASP.NET MVC. Enjoy.

Deleting a detached entity with EF4

2010-05-22T06:12:00.001-07:00

As unconventional as it may have seemed, it was the reality. With EF1 you had to first query the database to get the entity you wanted to delete. Talk about a waste of time, but the ObjectContext needs to be aware of the entity somehow. In my previous post I talked about how to update a detached entity with EF4. Now deleting a detached entity is just as easy.


  void DeleteBook(Book book)
  {
    using(var ctx = new BookEntities())
    {
      ctx.Books.Attach(book);
      ctx.DeleteObject(book);
      ctx.SaveChanges();
    }
  }

Again pretty easy and should improve on performance.

Updating a detached entity with EF4

2010-05-22T05:49:00.001-07:00

With the introduction of EF4 we now have the ability to update entities without having to query the database first. This was always a performance hit with EF1, but a necessary hit. I’m not going to get into that, lets see the code.


void UpdateBook(Book book) {
    using(var ctx = new BookEntities()){
        ctx.Books.Attach(book);
        ctx.ObjectStateManager.ChangeObjectState(
            book,
            EntityState.Modified);
        ctx.SaveChanges();
    }
}

Pretty easy. The drawback is that the ChangeObjectState method marks every property of the book object as being Modified. So when EF generates the SQL Update statement, every column in the database for this book record will be updated. This really isn’t an issue if you’ve set all the properties on the book entity. Where it becomes an issue is if you are trying to update only one property. For instance you just want to update book.Title, well if the Title property is the only property you set, then Author, Copyright, NumberOfPages, etc. will all get overwritten with null in the database.

Dream Team

2009-03-21T07:03:00.001-07:00

I've been developing software professionally now for about 7 years and for the past three I've been a consultant. I've met a lot of people, been on a lot of teams, and worked on some pretty cool projects. Recently I rolled off of a client project that was probably one of the most challenging projects I've ever worked on. My previous software development experience has been mostly with financial applications. This is helpful when trying to put the requirements for the next project into some sort of context. A loan is a loan and an interest rate is an interest rate. My last client was a Biotechnology company. I've never worked with highly scientific data and terminology before. Trying to put the project requirements into some familiar context was pretty difficult. The same level of difficulty though also made for a very rewarding environment. But that's not the point of this post. What I found most rewarding about the project was the team that came together. I was a member of a 5 person tech lead team. We had a collective experience that complemented each other better than any team I've ever worked on. Each of us could challenge the others in different ways. The dynamic of the team really forced you to think a problem through, and even then, when I felt I had a pretty solid design, they still poked holes in it. I can confidently say that I'm most proud of some of the components I wrote there. None of us knew each other before this project, and the Madison IT community is like small town America where everybody knows everybody, or so it seems. How we all ended up coming together is a mystery.

Face to Face with Employer

2009-02-23T16:29:00.001-08:00

So I'm an active member of Facebook. I love it. I've reconnected with several friends that I haven't seen in years and keep in touch with friends that don't live near by. I chat, write on my friends' walls, and have even planed reunions using Facebook. But recently people from my employer are now getting on Facebook and sending me friend requests. I have no problems with any of these people. I talk to them all the time at company get togethers and business meetings. But you won't find me out on a Friday night stomping the town with them. I like to draw a line between my personal life and my professional one. I feel I'm probably in the majority here. I'm not sure if I like the idea of management from my employer wanting to be friends with me on my Facebook. In contrast, if you are going to publish your life on the internet, then you better be willing to accept the fact that its there for everybody to read.

I digress.

Mentoring techniques

2008-11-11T07:37:00.001-08:00

Yesterday while at work I was asked by a junior developer what type of data access layer my team is using on our project. He was wondering about the use of data sets or business objects. Do we use inline SQL or stored procedures. So I sat with him and had a short conversation about his inquiries. After he left, I got to thinking, I better go see what he's working on before he tries to kill a mosquito with a canon. Good thing to. His initiative on his application was to take some ideas from a data access layer that another senior developer had wrote, and try and improve on it. I love the young sponges in the world. They just want to learn, push their abilities, and absorb as much knowledge as they can. But where do you draw the line with them? The data access layer he was going to try and enhance was solid, and has pasted the test of time. Its been implemented in several other applications as well. I'm all for enhancing and trying to find a better way, but the path he was going down I was skeptical about. It is my job after all to set him up for success. So I worked through some scenarios with him. I asked him how would you unit test your data layer? Would it be easier to maintain your design, or the current design? Does your design or the current design make more logical sense? My goal with him was to try and get him to understand the challenges that he'll face instead of telling him to just do it this way and be bombarded with a series of "what if's" and "ya but's". After he started to see the light, we talked about maybe some other technologies he could use instead of rolling his own data layer. One technology we talked about was the use of the ADO.NET Entity Framework. It basically builds the entire data layer for you, but its still cutting edge technology and maybe hasn't yet pasted the test of time yet. What other techniques do you use?

Scientific Notation to Decimal

2008-09-15T08:00:00.000-07:00

I recently had a situation come up where I needed to convert numeric data that was in scientific notation to a decimal. As an example I would receive a string that was in the form 1E-9. The decimal representation of this is 0.000000001. Seems simple enough.

string sVal = "1E-9";
decimal val = Convert.ToDecimal(sVal);

The above code fails because ToDecimal uses NumberStyles.Number which doesn't include the AllowExponent switch. You'll receive a FormatException exception that states the input string was not in a correct format. Use the following to convert to a decimal.

string sVal = "1E-9";
decimal val = 0M;

if(!Decimal.TryParse(sVal, NumberStyles.Float, NumberFormatInfo.InvariantInfo, out val))
val = 0M;

Jefferson

Mirra, Mirra, who's the fairest of them all.

2008-09-12T23:25:00.000-07:00

I've always wanted a nice home office. Back in December we bought a new house and I finally had the opportunity to set up a nice home office. Now that my office is on its way to being a sanctuary of productivity, I needed a nice office chair to finish the package. I shopped all over for office chairs. The the major retailers, and online. None really fit what I wanted. I had bought cheep chairs before, and they only really last a couple of years and don't support your body the way it should be. Then I started a new job, and the chair they provided was a nice ergonomic chair. Its a Hon. I finally thought I found the chair. I searched down a local retailer and went into their show room to check them out. Turns out this retailer does sell Hon chairs, but they are primarily a Herman-Miller dealer. I never really sat in a Herman-Miller so I tried out the Herman-Miller Mirra. The fit was right. The support was there. But that much money, and I wasn't sure how it is sitting in front of my desk with it. The selling point, they let me take home a trial Mirra chair for two weeks before I bought. Wow...lets do it. When I got the Mirra home, I immediately put it in front of my desk and I was sold. Hands down it is the best chair out there. After my two weeks was up I returned the trial chair and ordered one of my very own. Its a small investment, mine was $710.00. But if you spend a lot of time at your desk, its worth the investment. The construction is solid, the support is second to none. And it has a 12 year warranty. That right there pays for itself. If you're serious about getting a nice chair check out the Mirra.

Hello World

2008-07-27T20:22:00.000-07:00

This is my first post. I've been meaning to start blogging, but never got around to setting one up. What was holding me up? Trying to find a meaningful name. Something catchy, easy to for readers to remember. What best fits what I want to write about. Something all encompassing. Well maybe not all encompassing, but close enough. I've been a computer programmer professionally now for about 7 years. The latest bit of technology I'm reading about is a Microsoft product called Linq (pronounced Link). Linq stands for Language Integrated Query. If you are not a programmer, you won't find this interesting at all. But if you are, and you don't know what Linq is, check it out. Its great. Basically its a way to interact with several types of data sources all in a very common way. Linq is very extensible, and several flavors of it have been popping up all over the web. Linq to SQL and Linq to Xml are apart of .NET 3.5. Linq to Amazon, Linq to Active Directory, Linq to Oracle, Linq to SAP, some very cool stuff here. It hit me one night, Linq 2 You. Thats what blogs are anyway. Scribing my thoughts, for the world to read. This blog is my Linq 2 you.

Thanks for joining me,

Jefferson