Backup Options

Lea Alcantara:  This episode is sponsored by the Responsive Web Design Summit.  Check it out at rwdsummit.com.  Use the discount code EEPodcast for a 20% off your ticket and go beyond Google searches and those incomplete, out of date online tutorials.  Bring the experts to your desktop with the Responsive Web Design Summit, rwdsummit.com.

Emily Lewis:  The ExpressionEngine Podcast would also like to thank Pixel & Tonic for being our major sponsor of the year.  [Music ends]  Hi Lea, how are you doing?

Lea Alcantara:  Pretty good despite the fact that I just flew in from Montreal and got back at midnight.  [Laughs]

Emily Lewis:  Oh, ouch, a little tired this morning.  [Laughs]

Lea Alcantara:  Yeah, just a little bit, but traveling and all that fun stuff, I always enjoy doing that so it’s always worth it, the lack of sleep, I mean.

Emily Lewis:  Yeah, and Montreal is such a great city.  I visited there last July.

Lea Alcantara:  And so shout out to the Montreal EE-ers, I guess.  [Laughs]

Emily Lewis:  Yeah, yeah.  I got to meet Andrea DeMers and…

Lea Alcantara:  Nicholas?

Emily Lewis:  Right, Nicolas Bottari, and we hang out a little bit and chatted, so that was cool, but it’s such a great city.  There’s so much to do there, and a lot of good food which is always like the number one thing I’m looking for.  [Laughs]

Lea Alcantara:  Yeah, me too.  So before we introduce our guest today, why don’t we catch up on some EE news?

Emily Lewis:  Yeah.  There has been a bit of news coming out of the community.  First up is some news from EllisLab.  They’ve increased their support responses for their different support plans.  They are now guaranteeing in the Silver support plan first response in one business day instead of two.  The Gold plan promises four business hours instead of one business day, and Platinum promises one business hour instead of four.  So across the board, all plans now offer faster responses. 

Lea Alcantara:  That’s a really great move.  These numbers to me sound a lot more reasonable based on the fact that you’re paying for the support, and I also, I guess, like based on their blog entry, you understand why they might have launched with less favorable support just to see how well they can handle the new volume, perhaps.

Emily Lewis:  That makes sense, to move slowly.

Lea Alcantara:  Yeah, and then when they see like how great it’s working out, then great.  Let’s see how it works out when they have a faster support.

Emily Lewis:  Have you used the support yet since they’ve changed to this new plan model?

Lea Alcantara:  No, no, I haven’t.  For me it’s still Twitter and/or StackExchange at this point. 

Emily Lewis:  Well, and if any of our listeners are interested in trying out the support, EllisLab had offered a three free month, I guess, that’s a trial of the Silver support.  So you have until the end of this month, February 28th, to redeem that free three-month trial of Silver support.  So if you’re interested in trying it out, now is the time to do it. 

Lea Alcantara:  Well, it sounds good.

Emily Lewis:  Since you mentioned StackExchange, I think it’s worth mentioning it’s doing well and going strong, but it does still need support to come out of the public beta.  I was looking at the Area 51 staging area where it has some stats, and all of the five areas of the criteria that they decide when a site can go out of public beta were doing well on three of them, and were doing okay on two.  Where we need more is that we need more answers per day per question.  Right now, we are at 1.9% answer ratio and they are looking for 2.5, and then we need more visits per day.  Right now, we are hovering between 760 and 800, and what the StackExchange is looking for is about 1,500.  So get your questions on there, get answering.

Lea Alcantara:  Yeah, and I think Anna made a good suggestion as why not just make the StackExchange URL your home page when you open up your browser so that gives you an automatic visit and it also gives you a chance to see what questions are out there that you might be able to answer.

Emily Lewis:  Good idea.  I missed that news.

Lea Alcantara:  In other EllisLab official news, they have made their EE job board which used to be a part of their forums which is now just kind of community forums, not necessarily an official forum anymore, and they’ve moved it to Director-ee.

Emily Lewis:  I was looking at it.  It seems interesting.  I have to admit I’m not currently looking for work, so it’s not something I’m really paying a lot of attention to, but hopefully having an official forum will be helpful for job seekers and people looking for qualified ExpressionEngine devs.

Lea Alcantara:  Director-ee already has the infrastructure and people will have their names listed on there and everything like that.  So for those who are looking for some jobs, it’s kind of like the double tier.  They get to check out what developers have listed themselves there as well as what jobs are on there as well.

Emily Lewis:  So that’s at director-ee.com/jobs.  Oh, so a little news about Solspace.  This might be a bit late for some of our listeners, but Ryan Irelan decided to pass over the reins of the EE Help Chat, and Solspace took those reins and that’s now who’s hosting it.  It is at a new time than it was previously.  It’s now 4:30 Eastern for that Solspace EE Help Chat.  But they are also starting a new interview series focusing on business practices.  Mitchell talked to Matthew Weinberg for the first episode, and it was a great interview, and I’m really excited about this. 

Lea Alcantara:  Yeah, me too, like in the EE Podcast, I really am always happy when we talk about business practices like when we had Paul Burton here who talked about contracts, and when we had Brad Parscale who talked about sales and all that fun stuff, and so when Matthew was talking about his sales pipeline and how he’s dealing with all of that, I’m thrilled and always interested, and Erik Reagan kind of continues on in their next episode as well, which I haven’t had a chance to listen to yet, but it’s definitely on my radar.

Emily Lewis:  I’m excited about this too.  Just like you, I love the business practices’ focus, and I like Mitchell’s style as well.

Lea Alcantara:  I really like it too because they are so open about the struggles as well as the successes.  To be frank, it’s sometimes surprising to hear because Mitchell is so put together, and Solspace is such a big brand in EE community to have him say, “Well, maybe I should have done this more, and I did not.”  And I was like, “Wow, so even Mitchell has these issues.”  [Laughs]

Emily Lewis:  Yeah, well, that’s the kind of stuff that I think you learn the most from, so I’m so excited that he’s sharing it and they’re putting this out there. 

Lea Alcantara:  So let’s get started on our episode.  For the last one, we talked with Travis Smith about high traffic EE sites, and we briefly talked about backup options.  So today, we’re diving deeper into that topic with Eric Lamb of Mithra62.  Emily, why don’t you introduce Eric.

Emily Lewis:  Eric is an EE developer specializing in professional and enterprise grade add-ons including Backup Pro.  His company, Mithra62, is a leader in ExpressionEngine add-on development with a reputation for stability, usefulness and being highly configurable.  Eric is also half of the duo behind the new EE Developer Conversations on EE Insider.  Welcome to the show, Eric.

Eric Lamb:  Hey guys, how are you doing?

Emily Lewis:  Great, how about you?

Eric Lamb:  I’m doing good.  I’m doing real good.  I’m really excited to be here.

Emily Lewis:  We’re excited to have you on too, Eric.  Would you mind telling our listeners a bit more about yourself?

Eric Lamb:  Sure.  As you guys said, I’m Eric Lamb.  My company is Mithra62.  I’ve been doing ExpressionEngine add-on development for almost two years now.  It will be two years in June, and before that, I worked in great many agencies and just kind of floated around doing freelance to agencies that contract work to just straight on development.  I’ve been doing web development professionally for about 12 years now, almost 12 years.

Emily Lewis:  Wow, I didn’t realize that your time with EE was… I mean, two years is nothing to scoff at, but I assumed it was so much more just considering how much you have to say like, “You blog a lot about ExpressionEngine.  You blog a lot about your business practices.  You’re now sharing a lot of information on the EE Insider.  In fact, I read your agency post that you had up there the other day on EE Insider and that was really great.

Eric Lamb:  Yeah, I’ve been writing for years, years and years.  So when I started coming to ExpressionEngine, it was a very easy transition for me to just stop writing about normal everyday web development and kind of focus on ExpressionEngine, which is really what a lot of my writing was anyway, which is a bit of day to day diary of what I did.  It’s a bit of an extended memory for me.  I didn’t write so much to build out a library of articles.  They are more just to have a place where I could look and see how I did that thing tomorrow than I did three years ago.  It was more like an ongoing record.  So when I came to ExpressionEngine, it just carried over very naturally, but you are absolutely right, two years is still very, very low.  I’ve been very fortunate in just how quickly things have kind of taken off within ExpressionEngine for me.

Lea Alcantara:  I think that what you mentioned about how writing these articles was mostly so you can have a record of what you were doing.  It’s kind of like a more thoughtful documentation in some ways, which is actually pretty smart.  I think that more people, including myself, should be doing that just so you can see how things have grown and where things have gone, and I mean, that’s why I’m so happy that EE Insider exists because you can take a look at the breadth and the scope of how the community has changed and moved forward on top of just figuring out current events and the new technique or stuff like that.  I think sometimes documenting or writing articles or blogging, people talk about how it’s becoming a lost art, and I’m so happy that you are taking the time to write such long form information and discussion to really stir further discussion about specific topics.

Timestamp:  00:10:35

Eric Lamb:  Oh, definitely.  In fact, the writing I’ve been doing lately or ever since I got that column with EE Insider have been more thought provoking, more op-ed pieces, and not very technology focused, and the technology ones, those are really the no book style post.  They are the ones where it was just really for the next person who’s coming along who had this type of a problem.  They can kind of google through it and there it is.  But with what I’m writing about now, it’s more thought pieces, more about where I think the industry is, how it’s progressed, where we came from, and just trying to build out a discussion of more longer term idea.

Lea Alcantara:  In some ways, it’s kind of like, you see, this is how I’m going to try to put it back into our topic about backup options…

Eric Lamb:  [Laughs]

Lea Alcantara:  It’s like you’re having a backup of information.  [Laughs]

Eric Lamb:  [Laughs]

Emily Lewis:  Your idea backup.

Lea Alcantara:  Yeah, exactly, exactly.  Whenever I think about backup, I always think about worst case scenarios, and I think not enough individuals, myself included, take that as seriously, and it’s sort of weird because for me, backups are really, really important especially because I actually got my laptops stolen in the middle of a client meeting, and if I did not back anything up in two different places like I had a desktop computer as well as an external hard drive, I wouldn’t have been able to get right back to work the next day, which was what I was able to do because I had those options with me.  Emily, how about you?  Have you ever had any catastrophic backup issues?

Emily Lewis:  Yeah, in fact, Lea, you might remember September of last year I was offline for a while because I didn’t have my computer.  It was in the shop.  The 2006 Macbook I bought was my primary production computer for many years, and the first time the hard drive died, I did not have any backup in place.  It was kind of like my first Apple computer, like I’m just not a geek like that.  I’m like, “Backup, what?”  And so I learned quickly the hard way because I lost everything.  The only thing I was able to restore was some of my iTunes music from my iPod.

Eric Lamb:  [Laughs]

Emily Lewis:  And so I started using the time machine and I got an external hard drive, and then I got another Mac, and so I have those sort of backing up, but then the hard drive went once again, and all in all, that Macbook, the frigging hard drive died three times, but now I have backup practices in place.  The amazing thing is, and it amazes me because when Eric sent us all those links about you should be very scared if you’re not backing up, I started getting really scared and I was like, “How come I didn’t apply the lesson that I learned from my own personal data to my websites and my clients?”

Eric Lamb:  [Laughs]

Emily Lewis:  It’s like it completely disconnected, and it only started making sense to me when Eric sent those links, and I was like, “Oh my God, I need to talk to him today.  I can’t wait until next week.”

Lea Alcantara:  Yeah, exactly.  How about you, Eric?  Like have you always been concerned about backup options, or did a slap in the face kind of happened to you and you’re like, “Oh, I better deal with this because I’ve got bit really hard.”  When did you start backing stuff up?

Eric Lamb:  It was kind of just like you guys described.  It was a failure point.  I always paid a lip service like in school, that saying of, “Always save your documents as you’re writing it to just make sure,” and so small things like that, but it was basically just lip service.  I never really cared about it up until I needed it.

Emily Lewis:  [Agrees]

Eric Lamb:  And then when I needed it, I found out that I was doing it wrong, that I had basically just kind of painting myself into a corner and I couldn’t get myself out.  It was done really, really poorly, and it just kind of got me thinking of like, “Okay, well, there’s got to be a better way to do this.”

Lea Alcantara:  So what kind of issues were wrong?  What did you do wrong when you first started that you realized was like this was not the way to do it?

Eric Lamb:  My first introduction to doing backups on a professional level was using a tool that came bundled with Windows Server 2000, which was its own internal backup program, and I found out the hard way that that is an abysmal tool for backing up.  Well, for backing up, I guess it’s pretty good.  For restoring, it’s horrible.

Emily Lewis:  [Laughs]

Eric Lamb:  The reason being is that, and well, for one, it’s a Microsoft tool and we can point and laugh at that all day, but the other thing is that in order to restore it, it had to be done on the exact same server specs, exact same hardware, exact same everything.

Emily Lewis:  Oh.

Eric Lamb:  Or else you couldn’t actually restore, so if you have a cataclysmic server failure, which I did, then your backups were essentially useless, and that really kind of beat into me that proprietary formats within your backup software is just a really, really, really bad idea.  To really be optimal, you want to have direct access to those files, maybe a zip file would be about as hacked up as I would go, but the Microsoft backup products at that time, it was very proprietary and it was very much locked in to those server hardware specs, which is in hindsight, I just couldn’t imagine why somebody would think that that would be a good idea.  It bit me hard, and we ended up losing years and years of data, like the server died and our backups were all locked up, and essentially, we might as well has had no backups for as good as they did us.  So that really got me thinking about just how best to do it.  Even then, I mean, even after I recognized that and had my next hardware failure, I started realizing that there’s more to it than even that, which was we ended up having a problem where a server ended up dying, and in the process of restoring the server, it ended up [laughs], just through sheer incompetence on my part, deleting all of the backups.

Emily Lewis:  [Laughs]

Eric Lamb:  Like I had the backups set up to a separate drive, I’ve had everything on zip files where I could easily extract it.  Individual things, and thinking that I had it all set up, and then we left the drive in, so when we went to go and restored the server’s operating system, it ended up formatting the external hard drive.  [Laughs] Not my finest hour, but again, it was one of those like, “Okay, I mean, there’s got to be a better way to do this backup stuff,” and it’s just constantly been beaten into me over and over and over that you just really need to be as super careful as possible when it comes to your disaster recovery plan, and then there are very few things that are too far, like you’d feel like you’re going too far out of your way or being too paranoid.  In my experience, you are really not.  [Laughs]  I think there are ways for the universe to contrive against you, and it will.  It will actually come back to bite you.  So it’s just been a constant education thing on my part.  I think I have it figured out, and then I didn’t, and then I think I have figured it out again, and again I didn’t.  [Laughs]  So by this point, I feel like I really kind of figured it out.

Emily Lewis:  So let’s talk about this backup recovery plan.  Is it something that is just one big process, or is it something that you could take small steps and then add to it over time?  Like how do you approach it?

Eric Lamb:  They way I approach it now is pretty much how Backup Pro is built out to support.  Basically, you want to have your backups as spread out as possible.  Never having them on just one single hard drive or one specific network.  You want to have it in like for the way that I do my personal sites now is I have my server backups, which happens at the server level, which is just a dumb backup system.  It all does is it zips it up and it FTPs it into my home network.  But then this is not using Backup Pro, by the way.  This is using the server software. 

At the website level, I have Back Pro running for my ExpressionEngine sites and I have it FTP-ing the backups to my home network putting them up onto Amazon S3 for redundancy and also having them stored on an external hard drive on the Rack where my servers are at.  So I have three different copies, three different ways of data, and I just find that to be a load off my mind.  It really kind of keep things going to a point where I have the confidence that if one of those pieces fails, I can still get it from another way.

So like what had happened to me previously, just another personal anecdote, was I ended up having a client who basically just decided to rage quit the internet.  He was paying for a dedicated server, and he just decided, “No, I’m done.  I don’t want to do this anymore.”  He didn’t reach out to me.  He didn’t tell me anything, and I ended up losing all of my data, and what my backup plan at that point was to have everything stored locally on the server and then have everything stored on an external hard drive, and because he just rage quit the internet, I lost two different avenues of it. 

So when I started to conceiving it on Backup Pro, having a third layer of redundancy or even a fourth which actually Backup Pro has got all sorts of redundancies, then that become very, very crucial, just in case like what happens if Amazon decides, “We’re no longer going to do an S3.”  Well then, all my backups are gone.  But what happens if Amazon is just down for a time when I need to get that data?  I mean, a website being down for a day even is unacceptable, so with Amazon down, how can I make sure that I can access it?  Or what if my internal network ends up dying?  I still want to be able to make sure I can get them from Amazon or from Rackspace or from some other area. 

It’s very, very important to be able have access to your data, which sounds like a no brainer, but I got bitten by that quite a few times before I finally it out like redundancy is a great thing when it comes to backups.  You really just want to have them as all over the place as possible just to make sure that you don’t find yourself in that position of never being able to get your data back.  I mean, that’s the whole point of a backup.  So if you can’t get to the data, I mean, it’s just kind of a pointless thing.

Timestamp:  00:20:06

Lea Alcantara:  As you’re outlining all this, and it sounds amazing and perfect, and yes, it make sense, check, check, check, check, check, but there’s a couple of major issues that I can see right away.  The first issue, privacy and ownership of that data, and then the cost to be able to do this, and especially for client sites.

So let’s talk about the privacy issues first.  Right now, you’re mentioning you’re doing this for all your own personal sites and everything like that, but if you are doing all these levels for your clients, who owns that data, how do you deal with the fact that maybe specific sensitive pieces of files or data is being spread out to all these different services, especially like the larger your client, especially if you’re dealing with government or education, anything to do with children, even if the files don’t have anything to do with the kids just as long as it’s got any sort of data?  I’m just concerned that, yes, these are great for backup and things like that, but how do you deal with the privacy issues, and do you make your client sign something, or do they even understand that there’s a privacy issue?

Eric Lamb:  When it comes to my client sites, they do get a free copy of Backup Pro, because why not?  But when it comes to the actual services that they use, that is all up to them.  If they want to use Amazon S3, then they use it and they pay for it.  For me, I just prefer to have it spread out as possible just for my own well being.  But yeah, when it comes to the clients, I do give them the option to let them know, and I do, in fact, recommend that they set up at an Amazon S3 or a Rackspace account so that they can go ahead and have those backups being redundant.  If they have any concerns about it, which is yet to happen, but you do bring up a good point.  But if they did have any concerns about it, I would just say that they do have privacy policies in Amazon S3 and they do tend to make sure that your data is secure.  But it is an individual gut check thing.  It does kind of have to come up to you, but it doesn’t mean it just have to be that.  Like you can set it up with Backup Pro to have an FTP into your home network, which is another way that I use it.  If they didn’t want it with Amazon, they can still have it FTP-ed into their office.  So it stays on the server and then it gets FTP-ed into their home network.  In terms of security, and I’m glad you brought up the privacy because that’s when they segueway into security, you really do want to make sure that within your backups, you don’t have any credentials.  It’s just like with Version Control.  Again, I set all of mine with ExpressionEngine to not back up the database files so that the database credentials aren’t in there.  It’s certain things like that because it does save that in plain text, and if it does get compromised, well then your site is compromised.  But restoring to that database config files is very, very easy, so I look at that as a bit of an important piece.  You always want to make sure that you don’t have anything sensitive within the backups just in case.  But just a security best practices anyhow.

Lea Alcantara:  If that’s the case, then how do you make sure that you keep all those sensitive file data somewhere, like if you’re not backing those particular files up because of the plain text passwords or whatever?

Eric Lamb:  Well, the way that I look at it is you don’t really need to know what the password is.  You can always change the passwords.  You can reset the password, so having it written down somewhere is just as insecure as having it within your backups, like you never really… the password, but you can always change your passwords if you have system administration access, which most of us web developers do just going to cPanel to change their password, done and done.  That to me is a lot more secure and reliable than leaving them intact or trying to write it down and having it somewhere on like a sticky note or anything like that.

Because remember, most instances of hacking have something to do with social engineering, which is basically where somebody calls up where they ask you or they come into your desk and they just look for it.  Very rarely does a large scale hacking have very little to do with social engineering.  If somebody is going to really try and go after Lea, Lea specifically, social engineering would play in that.  The alternative is usually just like a script kiddie running something and trying to find unsecured stuff.  But if I’m going after Lea, I’m more than likely going to employ some social engineering techniques. 

The best way around that is to just have best practices when it comes to your passwords.  Don’t write it down.  Don’t have it anywhere somebody could actually come in and find it or ask the secretary to go to Lea’s desk and look at it.  So that carries over to backup strategy, I just prefer to keep as little sensitive data out of my backups as possible, just in case.

Lea Alcantara:  Why don’t we talk a little bit about the second issue that I noticed with all this redundancy, it’s the costs.  I mean, Amazon S3, I know is very inexpensive in general for data, but if you’re backing up a lot of files and then using that server, and then you’re using that for your clients and all that kind of stuff, and then you also have a server rack.  Those types of costs add up.  Would you mind letting us know what kind of costs that you personally are dealing with?

Eric Lamb:  I personally, I think I pay about 45 cents a month for my Amazon S3.

Lea Alcantara:  Oh wow!

Eric Lamb:  I mean, like it’s incredibly negligible.  Hard drive space is really cheap right now.

Lea Alcantara:  Oh wow!

Eric Lamb:  Three terabyte hard drive is for about a $150 now and the way that I deal with it, I mirror everything, so my home network where I have my backup is coming just through FTP.  I have two to three terabyte hard drives that are just set up at a RAID and they just mirror each other.  So if one of them die, I still have that info.  I mean, for $150 and having three terabytes, my in house network have backups of it as much as I can possibly get.  Amazon S3, as I mentioned, it’s less than a dollar.  It’s 45 cents I think was the last time I saw it, and even with Rackspace, it’s ridiculously cheap.  Hard drive space is very, very inexpensive.

Lea Alcantara:  Wow, I’m actually surprised by that because I think with the volume of backups that you’re doing, I guess I expected it to be more than a dollar at least, and 45 cents is kind of like almost of a no brainer.  Like if it’s going to be that inexpensive, of course, you’re going to use that system. 

Eric Lamb:  But one thing to keep in mind too is like I don’t have backups for everyday in history.  You see, the scheduling that I do with my stuff is I do database backup every day, so seven days a week, I have a database backup, and it’s a complete database backup, so basically a snapshot of everything up until any day from a week ago.  I’m sorry, now I keep two weeks’ worth of backups.  So each one of these, there’s probably in a compressed state, maybe 20 megs, and so we’re talking about like about 200 megabytes I’m actively saving.  That’s really, really inexpensive to keep.

And I do a file backup, a full file backup every week, which is extremely redundant because the files rarely change, or at least in my environments, I don’t do very much social networking stuff or user generated content.  But for me, just for my past experience where I’ve had catastrophic failure just in terms of missing data, I’d much rather have all of the data multiple times than none of the data when I need it.  But even with all of that, I mean, I rarely run into issues, and with my file backups, I think it comes out to about 200 megabytes per file backup with it’s all compressed, and I keep about 14 of those as well, and all of this is configurable within Backup Pro, like you can say, not to turn this into a commercial and I think Safe Harbor works in a very similar way, but I could be wrong, but you can go give it and only give it a gigabyte worth of space, or only keep up, let’s say, the last month’s worth of backup, 30 days’ worth or a gigabyte, but all of it is configurable so all of it is for me really, really hands off in terms of the process of it.  It just happens.  If I just set up a cron job, and it just goes, and it’s something that I never have to worry about.

Lea Alcantara:  Okay, and it’s interesting that it seems like you just go back as far as two weeks as the max.  Is that normal?  Like how do you determine how far back you should backup?  Because based on efficiency and it seems like a good practice that you’ve had was, “Okay, we’re going to back this up for one week and we’re going to back up the database every day, and we’re going to back up these other things for two weeks and then that’s it.  You mentioned 30 days, but how do you decide that?  How do you decide this needs one day, this needs two weeks, this needs one week, and this needs 30 days, and then anything beyond that, we’re not going to deal with, how do you decide that?

Eric Lamb:  Honestly, most of it is just kind of a gut thing.  It really just kind of come down to your specific situation and how your sites are going to run.  I mean, if you guys don’t get any problems, then really the backing up the database every day, you are not going to have very much content changing backing up the database every day.  It is probably not going to make nearly as much sense, but again, like if we go back to my previous experience with the pain of it, and considering how cheap storage is, I would just put it on to find out like I save 50 cents, but I don’t have any data.  I mean, considering the cost of it.  Even if it cost me $20 a month or even a $100 a month, being able to have that peace of mind, because if I understand like neither one of you guys have had to go through a major data loss, have you? 

Emily Lewis:  No.

Eric Lamb:  Well, I guess with the Mac.

Lea Alcantara:  Yeah, not on a major site yet, like in terms of I’ve had backup issues, but I usually did the old school stuff so then it’s like, “Oh-oh, I can just restore this manually slowly, but it was never catastrophic, like it never would have cost my client money.

Timestamp:  00:30:04

Eric Lamb:  There are very few fears in the world that I have like going to my client and telling them that their entire project is gone.  So I mean, just going through that, I mean, it’s just the agonizing, the panic you get in the pit of your stomach where you’re like, “You’ve got to get on the phone and just let them know like I’m really sorry guys.  I know you paid this absurd amount of money to build you up this thing and we had a hard drive failure.  I don’t know what to tell you.”  That’s a horrible situation to be in.

The reason why I tend to keep so many backups or so many days’ worth of backups is because I’ve been hacked and I didn’t even know it, and so trying to restore uncompromised files became a bit of a problem.  So if you get hacked and you don’t find out for a couple of days and all you have a day’s worth of backups, well, then your backups have hacked files.  They have compromised data.  So for me being able to go back two weeks, that feels like I mean I could never imagine not knowing what was going on with any one site over a two-week  time frame.  Like just having two weeks and not knowing anything that’s been going and then finding out they were hacked two weeks ago, I just don’t think that would happen with how I personally run my projects and how I personally look after my clients.  Having gone through that, again, it might as well storage is cheap.  Data is cheap.  So really, I mean, the arguments against it just don’t really add up to me personally.  I would much have it than not have it. 

Emily Lewis:  Well, so when you’re dealing with a client, let’s say, you’re working on the contract, you’re getting ready to sign, where does the backup and your data recovery plan fit into the contract?  Does that like a line item?  Is it something that it’s in there?  If the client pushes back and says, “We don’t want it.”  Well, obviously, you offer your clients your Backup Pro add-on for free, but if they don’t want to purchase an add-on, like how does that conversation go?  Is it what you’re saying here, or you’re just trying to convince your clients, or do they have a say and they could say, “No, we don’t want to do this for whatever our reason is,” would you walk away from the project at that point?

Eric Lamb:  Honestly, it’s non-negotiable for me. 

Emily Lewis:  [Agrees]

Eric Lamb:  This isn’t a situation where they can say no.  To me, I look at basically how I would look when my nephew is driving and he says he doesn’t want to wear his seatbelt.  This is non-negotiable.  You’re getting a backup strategy.  We’re going to make sure that your data is recovered.  Keeping in mind too that this only applies to clients that I personally host.  If they want to host elsewhere and they want to take the project and go cowboy and do whatever they want, that’s solely up to them.

But if I’m hosting it for them, they are absolutely paying for a data recovery plan, absolutely, and it’s not something that I charge extra for.  But really, I just need them to know that they need to pay for an Amazon S3 account or something redundant, and then FTP server somewhere, so we can go ahead and make sure that they have their data for them.  But I do not at all ever make it an opt in.  If somebody is trying to opt out of it, I just don’t even think that would be possible, that I could let them.

Emily Lewis:  Do you have clients that host elsewhere?  And if so, how do you work with them for backups on another host provider?

Eric Lamb:  It’s essentially the same thing.

Emily Lewis:  [Agrees]

Eric Lamb:  And yes, I do have clients who host elsewhere.  What I do is when I move it to their new server, I do set up everything, how I think it should go all set up.  I’ll let them know that they should have an S3 account or they should have a Rackspace.  Choose one, at least, and an external FTP, and then I do still take backups of their site to my own network just in case.

Emily Lewis:  [Agrees]

Eric Lamb:  But to me, I mean, because again space is cheap.  So I mean, even if on a three terabyte hard drive, they are a hundred megabytes worth of backups.  It’s not even going to dent the thing, so long as I’m providing support for them.  Like if they decide that they want nothing else to do with me after the project is over with, “Hey Eric, we’re done,” then no, I actually do not take backup for them.  I set it up as best as I can.  But if I’m on there in some sort of support thing, as support contractor, they are coming to me to modify their site every once in a while, I still do include them in their backup options. 

Lea Alcantara:  So we’ve been talking a little bit generally about backup options for these all these files and everything like that, are there any special considerations when backing up an ExpressionEngine site versus another system?  What are mission critical things you need to think about when backing up an ExpressionEngine install?

Eric Lamb:  There are certain files that are more important than others, but I tend not to think of it like that.  With Backup Pro, I backup everything because you set up a path, and you say, “Take this path and give me everything in that path, just backup.  This is what I’m going to backup.”  Then you can add another thing and say, “Only backup this directory, this directory, and another directory, and exclude out this other stuff.” 

But I really do, for me, I just set it to take the entire HTML directory or the HT docs or whatever at the website is, and just backup everything.  There are very few reasons not to do that outside of they have an HT docs which is just full of like hundred megabyte video files or things like that.  Those surely you don’t need to backup outside of the ones.  I don’t personally think that there’s very much of it in an ExpressionEngine website and every other website out there when it comes to backing it up.  You want the files, you want the database, and once you have these things, you are pretty much good.  I mean, you could look at it as like a case by case thing where it’s certain things and depending on how it’s set up, you want to grab some other files from elsewhere, but I always just say, “Grab everything.  Backup everything.”  Like I keep saying, space is cheap.  It cost very little for you to grab everything as opposed to being hit again, and again, it’s better to have it and not needed than to need it and not have it.  So grab everything.  Always take it all.

Lea Alcantara:  Since you created Backup Pro, clearly you’ve got all these options to do all these different types of backups.  I’m curious that you’ve only been working with the EE community and ExpressionEngine for just a couple of years.  So what were some of the things you did before you developed Backup Pro?  Or was Backup Pro something you just started doing right away and did use personally before you sent it out, or did you just use old scripts?  What did you do before you created Backup Pro?

Eric Lamb:  Before I created Backup Pro and started using it for ExpressionEngine, I did my backups the same way a lot of people do.  I shell script it basically.  It’s just a small little shell that show you how it will run on a cron and it’s basically how ExpressionEngine work with one key difference, which was on incrementals.  I’m not a big fan of the incremental backups for specifically the reasons I laid out earlier.  It just makes things hard to restore.  If any piece of data is off, then your incremental backups will fail.  So I would much rather just have everything.  I used to do everything using a simple shell script, and I would just have that run off of cron and just kind of handle everything on its own. 

But that doesn’t really scale all that well, and once you get outside of having root access to a server, shell script is not just very practical.  Backup Pro isn’t the first backup program I’ve written.  One was when I was working for agencies and stuff, I had written a little backup thing for our in-house programs just to make sure that that we always had access to the data.  Backup Pro is most certainly wasn’t the first one I wrote, and in fact, I think the third backup stuff I specifically wrote, in fact, the backup engine itself or how it determines the zip files and how to backup the database, all of that I ended writing that library years ago, and just when it came time to do it for ExpressionEngine, it was a very easy fit.

Emily Lewis:  So Eric, you’re talking about some of the add-ons you’ve developed for backups, but if you don’t mind, I’d like to take a step away from backups and just talk about add-on development in general.  I was reading your Mithra62 blog and you had decided to offer developer licenses for Backup Pro, and then after about a year of doing that and evaluating it, you decided to stop offering that developer license option.  Can you talk about why you tried the developer license option, and then why you decided it wasn’t a good fit for you?

Eric Lamb:  I started doing the developer licenses frankly because a lot of people were asking me for it.  It almost a kneejerk where you have released an add-on, the reception will be very encouraging, but then there will always be somebody asking you for a developer license.  For me, I kind of bought into it at first.  Like I really thought, “This is great.  Like why wouldn’t you want to offer this?”  But then after talking to a whole bunch of other developers, it kind of became clear that at least to me in the way that I worked at, I was kind of shooting myself in the foot long term.  It was strictly kind of selling out the future for today, and as I think I laid out in the article.  I think that’s how I phrased it.  I’d make a little bit of profit today, but the customer base within ExpressionEngine is almost too small to support that model long term. 

So I ended up selling a decent amount of developer licenses, but for things that we’re using at hundreds of times.  In fact, what really hit this home for me was at EECI by just talking to a few customers, and they kept telling you that they really love the product, and they bought the developer ones and they are using it literally on hundreds of different websites, that they wouldn’t have had no problem paying for the single license, that they love it so much, but I offered up the developer license, and so they got the developer license, and that just kind of really gotten me thinking like, “You know, for that price, I’m literally leaving all this money on the table.”

You could break it down to a money phase, but the amount of work that goes into these things is not trivial.  It’s hundreds and thousands of hours and to have to continually support the same recurring issues with the same customers even though they bought one license, it just became demoralizing almost.  It was like working for free almost, and it just kind of kept things from being as fun as they should.  So I started thinking about it.

Emily Lewis:  [Agrees]

Timestamp:  00:39:44

Eric Lamb:  I mean, to hear from the customers that they didn’t mind paying a single license, especially for things that they use a lot, it just seemed to make a little bit more sense, so I thought I’d try the experiment.  I’ll just go ahead and cut these off and see how it goes, and I don’t think that it has been a success at all.  In fact, it may even be I made a bit of a mistake with killing up the developer licenses revenue wise.  Long term it still feels kind of right.  I don’t know if I’m going to go back to it, I might.  If you’re wondering if it has affected revenue, it almost definitely has, but not enough to make me want to open those back up.  In fact, the flipside though is it’s also really affected support in a good way.  In that I basically have almost no support request coming in for those specific add-ons, and when they do, it’s a very easy one on one type thing.  But it just kind of got to a point where I just started to question, “Was this worth it?”  Like, “Am I kind of putting the cart before the horse almost?”

Lea Alcantara:  Interesting.  Because for me as a consumer of ExpressionEngine add-ons, yes, it’s great to have developer license so I just pay one thing and then I don’t have to put it as a line item for clients and then I just take whatever is left and just charge that my clients instead, but for me personally, I always tell my clients if they want more, they need to pay more.  So if they need these specific types of features, then they need to fork the money out for that, and sometimes, I guess it’s frustrating when you are a developer. 

I mean, I was chatting with a friend of mine about this, how it’s like you know ExpressionEngine can do better.  You’re like you just know that it can and that you can give this value to your client and it’s going to be amazing, but you also have to deal with the reality of time constraints and their budget, and you can’t go beyond the scopes simply because you know it can do better.  You have to be able to sustain your own business and then sustain other people’s businesses so they can continue to give you these good solutions as well.

At the same time I know that at least in my experience, that the moment that money is exchanged with clients and that there are specific lines that are drawn saying, “Yes, this is included with the price.  No, this is extra and you need to pay for this extra things because that’s out of the original scope that we discussed,” then there’s value added to that as opposed that they just think, “Oh well, just throw it in because it’s ‘easy,’” or something like that. 

Most clients don’t know extensive the difficulty or the things that we have to do in order to make certain things done.  They just ask for the feature and have no clue how that’s even executed.  So sometimes I think that having the separate licenses show that, yeah, this isn’t something that is normally added in normal cases, or different cases, or whatever.  That it’s specific to your project.

Eric Lamb:  Another thing that kind of influenced the developer license issue was Backup Pro(ish), the free version, where that scenario that you just kind of laid out where you trying to line item things.  It seemed like with Backup Pro(ish) being out there, that there’s the need to have developer license in order to protect our client sites was kind of moot compared to – I mean, you can do it – in fact, Safe Harbor has a free one as well.  You can easily backup all of your stuff for free, but there’s no reason to actually have a developer license or to buy it in bulk.  If you just want to have a basic backup thing, you’re more than covered.  When it came to the developer licenses, that definitely applied.  Actually, I find it interesting that you guys line item out the add-ons that you sell?

Lea Alcantara:  For me, it depends.  I don’t line item when I do kind of like a scope, and then I just explain, “Here’s the price.”  I line item when they start asking me for stuff that wasn’t part of the scope.  Because basically, it’s just like, “Well, that wasn’t part of our discussion.  That wasn’t part of the functionality you asked.  It’s going to be this much, plus like the labor, of course, right?” 

Eric Lamb:  Okay, yeah, that actually makes a lot more sense.  I was curious about that, but that makes perfect sense.  If they are going to hit you up with creep scope, like yes, we should talk about that.

Lea Alcantara:  Yeah, exactly.  Otherwise, for me personally, I just say, “Okay, we have our discussion.  We talked about what the project is all about.”  And then I just give them a price and say, “Yes, this includes these particular functionality and licenses for these particular pieces of software,” which I decided would be needed based on our discussion, and then when they start saying, “Oh, I want this gallery, or I want this other thing,” that they think is “easy,” then I say, “No, it is not easy.”  [Laughs]  “And here’s how much extra it will cost you.”

Eric Lamb:  Yeah, that is frustrating.  I’ve never figured out a way to be able to just let scope creep off my back.  It’s still in theory for me every time.

Lea Alcantara:  [Laughs]

Eric Lamb:  And you’re surprised it’s going to cost money?

Emily Lewis:  So Eric, one last question before we let you go.  I wanted to bring up the EE Developer Conversations you’re doing with Ben Croker.  So for our listeners, this is a bit of like it’s an informal conversation between you and Ben.  You both are developers, and you talked about your own experiences and about being a developer working with add-ons and such.  How did this idea come about in terms of sharing this information on EE Insider?

Eric Lamb:  That was all Ben.  Ben and I have been pretty good friends for a couple of years now.  In fact, he was one of the first fans with Backup Pro.  He and his company, in fact, had paid for some added development, the Rackspace Cloud file feature that it has. Vinveo paid for it all. So Ben and I had met years ago, and just mutual respect between the two of us, and we just started having these little discussions.  Ben decided that he was going to kind of record one, and when they asked me about it at first, I had no expectation.  It is definitely Ben’s brainchild.  He’s the one who does the editing of it. 

It just kind of came about organically.  We don’t really plan it.  In fact, I probably even shouldn’t be talking about it because that’s one of the things.  I mean, it is so minimal in terms of Ben and I, how we look at it.  It’s just something that we do.  We don’t really plan on a show so much, but he and I talk all the time.  Ben will just randomly record it and then I’ll get an email saying, “Oh, here is the show.”  I’m like, “Okay, cool, right on.”  And then I’ll write up a little thing.  But it’s very much something that just kind of happened.  It’s not anything that we really planned.  The reception has been very, very encouraging.  In fact, I’m kind of sold to see more people doing it.  As you guys mentioned…

Lea Alcantara:  Mitchell and…

Eric Lamb:  I totally flaked on Mitchell’s name.  But Mitchell…

Lea Alcantara:  Yeah.

Eric Lamb:  And Matt and Eric.  But it’s just something that has kind of happened.  It just kind of came about.  It’s certainly not anything that’s a big thing.  In fact, he and I never really talked about actually doing one up until it happened.

Lea Alcantara:  In some ways, it’s like when I listened to the Developer Conversations, it’s kind of like listening in like almost eavesdropping at EECI.  [Laughs]  Do you know what I mean?  As in like you just ran into somebody like at the conference and you just started talking and then you’re giving a sneak peek over your conversation. 

Eric Lamb:  In fact, it’s funny you mentioned EECI because I think that was the catalyst.  Ben and I ended up having a really deep conversation about just random programming stuff in ExpressionEngine in general, and then Ben ended up hitting me up about it.  He was like, “Hey, would you mind if we, you know, started recording these things?”  I’m like, “Yeah, I don’t care.”  But I do think it was EECI where Ben just kind of got this idea of like let’s just start recording these things, which I thought was great.

The big thing that I like about it is that it kind of demystifies a lot of the stuff.  In fact, one of you guys hit it on the head perfectly, mentioning about how it was so interesting to see that Mitchell has these same doubts, who runs in through these same problems, or sees that things don’t work, and with programming, it’s very much true, and all of us kind of have had the same issues, the same questions, and we kind of beat up on ourselves a little bit where it feels very alone, nobody knows what I’m going through.  But we are all going through the same thing, and it was just we’re like, “Well, let’s just kind of break down this small little wall in terms of letting people know that we are human, that we all make the same mistakes, that this stuff is hard for all of us, and even if it looks like somebody knows what they’re doing, we’re failing 90% of the time anyhow.” 

Going to Ben and talking about this stuff and balancing ideas off of each other, that’s how we kind of figure this stuff out.  A lot of people just see the end result where it’s just like, “Oh, look at that super smart person who did this thing.”  But the reality is not so much.  I mean, there’s a lot of trial and error and a lot of discussions.  But just trying to make sure that that people know this stuff is hard.  It’s hard for everyone.  It’s hard for you.  It’s hard for me.

Lea Alcantara:  Well, we’re all human, like at the end of the day, we’re all human. 

Eric Lamb:  I mean, we’re all trying to figure this out just like everyone else, and it may look like we know what we’re doing, but nothing could be further from the truth.  We’re trying just like everyone else.  So for me, that’s when one of the big takeaways I take from it.  Like when I was first starting out, I would have loved to have heard a couple of developers talking about this stuff like Jeff Atwood or Joel Polsky just actually kind of letting out the realities of the process instead of just seeing the end result where you’re just like, “Wow, these people are so much smarter than I am.”  And you’re just like, “No, they are really not.”  [Laughs]  They are human just like everyone else, so I am kind of big fan of that aspect of it.  I don’t know where Ben falls on it, but we rarely actually talk about it as a thing.  We just talk and then we put it out.   For me, that’s the big takeaway that I personally enjoy, just letting people know that we’re not as smart as people think we are, or it’s not as easy as it looks.

Lea Alcantara:  Well, Eric, this has been a great conversation.  We have gone on very long.  [Laughs]  So I think we should cut it off there.  Thanks for joining us, Eric.

Eric Lamb:  Thank you guys for having me.

Emily Lewis:  It was great talking with you.  In case our listeners want to follow up with you, where they can find you online?

Eric Lamb:  Mithra62.com is where my add-ons are showcased, and @Mithra62 on Twitter is where you can find me to just have a nice little conversation. 

Lea Alcantara:  Perfect.  [Music starts] So now, we’d like to thank our sponsors for this podcast, Responsive Web Design Summit and Pixel & Tonic.

Emily Lewis:  We also want to thank our partners, EngineHosting, Devot:ee and EE Insider.

Lea Alcantara:  If you want to know more about the podcast, make sure you follow us on Twitter @eepodcast or visit our website, ee-podcast.com.

Emily Lewis:  Don’t forget to tune in to our next episode when Jessica D’Amico will join us for our get to know #eecms series.  Be sure to check out our schedule on our site, ee-podcast.com/schedule for more upcoming topics.

Lea Alcantara:  This is Lea Alcantara.

Emily Lewis:  And Emily Lewis.

Lea Alcantara:  Signing off for the unofficial ExpressionEngine Podcast.  See you next time.

Emily Lewis:  Cheers.  [Music stops]