freebsd, quick deployments, shell scripts

At work, I support three operating systems right now for ourselves and our clients: Gentoo, Ubuntu and CentOS.  I really like the first two, and I’m not really fond of the other one.  However, I’ve also started doing some token research into *BSD, and I am really fascinated by what I’ve found so far.  I like FreeBSD and OpenBSD the most, but those two and NetBSD are pretty similar in a lot of ways, that I’ve been shuffling between focusing solely on FreeBSD and occasionally comparing at the same time the other two distros.

As a sysadmin, I have a lot of tools that I use that I’ve put together to make sure things get done quickly. A major part of this is documentation, so I don’t have to remember everything in my head alone — which I can do, up to a point, it just gets really hard trying to remember certain arguments for some programs.  In addition to reference docs, I sometimes use shell scripts to automate certain tasks that I don’t need to watch over so much.

In a typical situation, a client needs a new VPS setup, and I’ll pick a hosting site in a round-robin fashion (I’ve learned from experience to never put all your eggs in one basket), then I’ll use my reference docs to deploy a LAMP stack as quickly as possible.  I’ve gotten my methods refined pretty well so that deploying servers goes really fast — in the case of doing an Ubuntu install, I can have the whole thing setup close to an hour.  And when I say “setup” I don’t mean “having all the packages installed.”  I mean everything installed *and* configured and ready with a user shell and database login and I can hand over access credentials and walk away.  That includes things like mail server setup, system monitoring, correct permissions and modules, etc.  Getting it done quickly is nice.

However, in those cases of quick deployments, I’ve been relying on my documentation, and it’s mostly just copy and paste commands manually, run some sed expressions, do a little vim editing and be on my way.  Looking at FreeBSD right now, and wanting to deploy a BAMP stack, I’ve been trying things a little differently — using shell scripts to deploy them, and having that automate as much as possible for me.

I’ve been thinking about shell scripting lately for a number of reasons.  One thing that’s finally clicked with me is that my skill set isn’t worth anything if a server actually goes down.  It doesn’t matter if I can deploy it in 20 minutes or three days, or if I manage to use less memory or use Percona or whatever else if the stupid thing goes down and I haven’t done everything to prevent it.

So I’ve been looking at monit a lot closer lately, which is what I use to do systems monitoring across the board, and that works great.  There’s only one problem though — monit depends on the system init scripts to run correctly, and that isn’t always the case.  The init scripts will *run*, but they aren’t very fail-proof.

As an example, Gentoo’s init script for Apache can be broken pretty easily.  If you tell it to start, and apache starts running, but crashes after initialization (there’s specifics, I just can’t remember them off the top of my head) the init script thinks that the web server is running simply because it managed to run it’s own commands successfully.  So the init system thinks Apache is running, when it’s not.  And the side effects from that are that, if you try to automatically restart it (as monit will do), the init scripts will insist that Apache is already running, and things like executing a restart won’t work, because running stop doesn’t work, and so on and so forth.  (For the record, I think it’s fair that I’m using Apache as an example, because I plan on fixing the problem and committing the updates to Gentoo when I can.  In other words, I’m not whining.)

Another reason I’m looking at shell scripting as well is that none of the three major BSD distros (FreeBSD, NetBSD, OpenBSD) ship with bash by default.  I think all three of them ship with either csh or tcsh, and one or two of them have ksh as well.  But, they all have the original Bourne shell.  I’ve tried my hand and doing some basic scripting using csh because for FreeBSD, it’s the default, and I thought, “hey, why not, it’s best to use the default tools that it ships with.”  I don’t like csh, and it’s confusing to try and script for, so I’ve given up on that dream.  However, I’m finding that writing stuff for the Bourne shell is not only really simple, but it also adds on the fact that it’s going to be portable to *all* the distros I use it on.

All of this brings me back to the point that I’m starting to use shell scripts more and more to automate system tasks.  For now, it’s system deployments and system monitoring.  What’s interesting to me is that while I enjoy programming to fix interesting problems, all of my shell scripting has always been very basic.  If this, do that, and that’s about it.  I’ve been itching to patch up the init scripts for Gentoo (Apache is not the only service that has strange issues like that — again, I can’t remember which, but I know there were some other funky issues I ran into), and looking into (more) complex scripts like that pushes my little knowledge a bit.

So, I’m learning how to do some shell scripting.  It’s kind of cool.  People always talk about, in general, about how UNIX-based systems / clones are so powerful because of how shell scripting works .. piping commands, outputting to files, etc.  I know my way around the basics well enough, but now I’m running into interesting problems that is pushing me a bit.  I think that’s really cool too.  I finally had to break down the other day and try and figure out how in the world awk actually does anything.  Once I wrapped my head around it a bit, it makes more sense.  I’m getting better with sed as well, though right now a lot of my usage is basically clubbing things to death.  And just the other day I learned some cool options that grep has as well, like matching an exact string on a line (without regular expressions … I mean, ^ and $ is super easy).

Between working on FreeBSD, trying to automate server deployments, and wanting to fix init scripts, I realized that I’m tackling the same problem in all of them — writing good scripts.  When it comes to programming, I have some really high standards for my scripts, almost to the point where I could be considered obsessive about it.  In reality, I simply stick to some basic principles.  One of them is that, under no circumstances, can the script fail.  I don’t mean in the sense of running out of memory or the kernel segfaulting or something like that.  I mean that any script should always anticipate and handle any kind of arbitrary input when it’s allowed.  If you expect a string, make sure it’s a string, and that it’s contents are within the parameters you are looking for.  In short, never assume anything.  It could seem like that takes longer to write scripts, but for me it’s always been a standard principle that it’s just part of my style. Whenever I’m reviewing someone else’s code, I’ll point to some block and say, “what’s gonna happen if this data comes in incorrectly?” to which the answer is “well, that shouldn’t happen.”  Then I’ll ask, “yes, but what if it *does*?”  I’ve upset many developers this way. :)  In my mind, could != shouldn’t.

I’m looking forward to learning some more shell scripting.  I find it frustrating when I’m trying to google some weird problem I’m running into though, because it’s so difficult to find specific results that match my issue.  It usually ends up in me just sorting through man pages to see if I can find something relative.  Heh, I remember when I was first starting to do some scripting in csh, and all the search results I got were on why I shouldn’t be using csh.  I didn’t believe them at first, but now I’ve realized the error of my ways after banging my head against the wall a few times.

In somewhat unrelated news, I’ve started using Google Plus lately to do a headdump of all the weird problems I run into during the day doing sysadmin-ny stuff.  Here’s my profile if you wanna add me to your circles.  I can’t see a way for anyone to publicly view my profile or posts though, without signing into Google.

Well, that’s my life about right now (at work, anyway).  The thing I like the most about my job (and doing systems administration full time in general) is that I’m constantly pushed to do new things, and learn how to improve.  It’s pretty cool.  I likey.  Maybe some time soon I’ll post some cool shell scripts on here.

One last thing, I’ll post *part* of what I call a “base install” for an OS.  In this case, it’s FreeBSD.  I have a few programs I want to get installed just to get a familiar environment when I’m doing an install: bash, vim and sometimes tmux.  Here’s the script I’m using right now, to get me up and running a little bit.  [Edit: Upon taking a second look at this — after I wrote the blog post, I realized this script isn’t that interesting at all … oh well.  The one I use for deploying a stack is much more interesting.]

I have a separate one that is more complex that deploys all the packages I need to get a web stack up and running.  When those are complete, I want to throw them up somewhere.  Anyway, this is pretty basic, but should give a good idea of the direction I’m going.  Go easy on me. :)

Edit: I realized the morning after I wrote this post that not only is this shell script really basic, but I’m not even doing much error checking.  I’ll add something else in a new post.

# * Runs using Bourne shell
# * shells/bash
# * shells/bash-completion
# * editors/vim-lite

# Install bash, and set as default shell
if [ ! -e /usr/local/bin/bash ] ; then
	echo "shells/bash"
	cd /usr/ports/shells/bash
	make -DBATCH install > /dev/null 2>&1
	if [ $? -ne 0 ]; then
		echo "make install failed"
		exit 1
	echo "shells/bash - found"
if [ $SHELL != "/usr/local/bin/bash" ] ; then 
	chsh -s /usr/local/bin/bash > /dev/null 2>&1 || echo "chsh failed"

# Install bash-completion scripts
if [ ! -e /usr/local/bin/ ] ; then
	echo "shells/bash-completion"
	cd /usr/ports/shells/bash-completion
	make -DBATCH install > /dev/null 2>&1
	if [ $? -ne 0 ]; then
		echo "make install failed"
		exit 1
	echo "shells/bash-completion - found"

# Install vim-lite
if [ ! -e /usr/local/bin/vim ] ; then
	echo "editors/vim-lite"
	cd /usr/ports/editors/vim-lite
	make -DBATCH install > /dev/null 2>&1
	if [ $? -ne 0 ]; then
		echo "make install failed"
		exit 1
	echo "editors/vim-lite - found"

# If using csh, rehash PATH
if [ $SHELL = "/bin/csh" ] ; then

pear list

I’ve been tinkering with PEAR at work, switching between using portage to install stuff and sometimes using pear directly to install it.

One thing that’d be nice is to get a list of the packages installed in pear command-line syntax. I.e. pear install MDB2-beta.

So, here’s a quick reference to convert the output of “pear list” to a list you can use with pear:

pear list | egrep "(stable|beta|alpha)$" | while read line; do echo $line | cut -d " " -f 1,3 --output-delimiter=-; done

A sample output would be:

$ pear list
Archive_Tar 1.3.7 stable
Auth_SASL 1.0.4 stable
Console_Color 1.0.3 stable
Console_Getopt 1.2.3 stable
Console_Table 1.1.4 stable
Crypt_HMAC 1.0.1 stable

to this:

etc …

For me it’s just a nice way to backup the pear module list, or copy it to a file and then install the pear modules on another box.

uphpu presentation: ripping dvds

I gave a short presentation last night at uphpu on ripping and encoding DVDs under Linux. I wanted to type up a few notes on some stuff I mentioned during the meeting.

Short version: Use Handbrake (available on Linux, Mac OS X and Windows). I’ve gone through a lot of applications through the years trying to find that “perfect rip”, and Handbrake beats everything hands down. I’m a videophile who likes everything looking as nice as it can, and this is what I use regularly.

Accessing DVDs

The first things you’ll need installed on your box is libdvdread and libdvdcss. libdvdread will access the DVD filesystem, and libdvdcss will decrypt the DRM on the disc. These should both ship with your distro, or if they don’t, should be relatively easy to acquire.

I use a lot of command line tools to access DVDs, so my focus was on that. The reason for that is because I have a lot of shell scripts to rip my massive library of discs, and I store data about them in a database.

lsdvd is the first stop. A very small app that uses libdvdread to display human and script-friendly output of the details of your disc.

mplayer dvd:// and mplayer dvdnav:// plus the optional track (dvd://3) or optional -dvd-device argument to specify the location of the device or ripped ISO.

I use dd to copy the contents of the entire disc to the harddrive and then rip it from there, to avoid wear and tear on the DVD drive, and also because I can do things like simultaneously rip one disc in the background and encode another at the same time.

When accessing the drive, you’ll want to use mplayer or some other program to access the drive and decrypt the CSS so that the drive itself caches the access. Otherwise, it may lock up.

I use pv to dump the disc and give me a pretty progress bar with ripping ETA and disc read speed: pv -ptre -w 80 /dev/dvd | dd of=movie.iso

If you want to extract the chapters, you can use dvdxchap, which is part of the ogmtools package.

DVD Tracks

Every movie I’ve run into has one track specifically for the video. Using lsdvd you can find the longest track, time-length wise, and that will be the movie.

TV shows on DVD often put many episodes into one track, where one or more chapters may contain an episode. You’ll need to examine these yourself to find out where each one starts and each one ends.

Another caveat is that they are not always ordered on the track or on the DVD in the same order that shows up in the menu. The best advice when accessing tracks and titles on DVDs is to never assume anything, if you are trying to preserve order.

DVD Subtitles

Subtitles on DVDs come in two formats: VobSub and Closed Captioning. VobSub subtitles are images that are overlaid onto the picture. Closed captioning is a text format.

If you have a ripped VOB (DVD MPEG-2 video), you can extract the subtitles using a program called ccextractor. It will save the subtitles to SRT, a subtitle format that can be muxed into your final video.

Encoding DVDs

Handbrake is a very “smart” application that will make things extremely easy on you. It supports encoding using three codecs: x264, ffmpeg and theora. x264 is an actively-developed H.264 video codec, and is used professionally by many companies. It also has a lot of options that you may find useful in your encode, based on your target playback device (ipod, ipad, iphone, apple tv, htpc, computer, etc.).

Handbrake supports muxing (or putting audio and video into a wrapper format) to two containers: MP4 and Matroska. MP4 is a very popular standard used in lots of places. It’s an open-specification. Audio is usually AAC. Matroska is an open-source openly developed container format that has a standardized specification. Support for it is growing in a lot of devices. Matroska can handle multiple video streams, multiple audio streams, multiple subtitles, stores metadata, chapters and attachments. Strictly speaking, either one will work fine for most people’s needs, but for those looking for advanced usage and tagging, Matroska will fit your needs. On linux, the tools for muxing and accessing information about a Matroska file are found in the mkvtoolnix package.

Deinterlacing and Decombing

Most movies will be progressive video, meaning that each frame is one full picture. Production studios and DVD authors may have their video interlaced. Interlaced video is where half of the frame is shown in quick succession where the second half is shown. On older TVs (like the CRT tube TVs) display at a native resolution of 480i, so you won’t notice the lines. Progessive displays (computer monitors, HDTVs, etc.) will display the lines and it can be a visual annoyance.

Handbrake uses filters to convert interlaced video to progressive video, called deinterlacing. Decombing is also a feature that Handbrake supports, which is basically “smart” deinterlacing. It is safe to turn on the decombing filter to the default setting for all your encodes, and let Handbrake automatically convert your video for you. Don’t use the deinterlacing filter unless you specifically know what you are doing.

Please see for a quick explanation of video deinterlacing.

Other Handbrake Features

Handbrake has lots of handy features. It has support for “profiles” which is a combined set of preferences for container format, video codec, audio codecs, preferred subtitle settings, post-processing, deinterlacing, decombing, and all the H.264 settings that come with x264.

One nice thing it will do as well is auto-crop your feature film for you. This is a boon if you have a letterbox source video, where the movie itself is in widescreen, but the presentation is in fullframe.

You can also “queue” up your encodes, so Handbrake can be working on multiple files. You don’t need to wait for one to finish before assigning it it’s next task.

Handbrake is also multi-threaded when using x264 to encode your video. With a six-core desktop, I have been able to encode video using the “Normal” profile between 160 to 180 frames per second. Meaning a 25 minute video will take about 2 1/2 minutes to encode.

PHP Classes

I’ve got some small classes that I use in my shell scripts for Matroska and Handbrake (v0.95 required). You may or may not find these helpful. I should warn that these are just used on my own set of scripts, so they may or may not be very clean or can have small bugs.


If you want more information, here are some good places to go:

Multimedia wiki
Handbrake forums

Also, feel free to contact me if you have any questions, and I’ll be glad to help.

wrapper script for disc_id

I wrote a little wrapper script for disc_id tonight, available here. disc_id is a little binary that ships with libdvdread, or at least, it used to in older versions.

I use disc_id to give me a unique 32-character string of a DVD, so I have an identifier to track them by in my database of DVDs.

I don’t know if it’s just me or not, but my DVD drives have issues trying to poll the devices. Once I insert a disc, it will take a few seconds for it to register completely so I can access it. However, binaries that access it will think it’s ready to respond sooner than it is able, and will die unexpectedly. So what I needed was a way to get the disc id and not worry about whether or not the drive has finished registering or not.

I just call my little script dvd_id and it is simply a small wrapper that checks the exit code of the disc_id binary. If it doesn’t work the first time, it sleeps for one second and tries again, then repeats the process until it gets a successful exit code of zero.

That’s it. Pretty simple, but like all little scripts, you really tend to depend on them.
if [[ -z $DEVICE ]]; then

if [[ ! -b $DEVICE ]]; then
echo "Device $DEVICE doesn't exist" >&2
exit 1

while [[ $EXIT_CODE != 0 ]]; do
/usr/local/bin/disc_id $DEVICE 2> /dev/null

if [[ $EXIT_CODE != 0 ]]; then
sleep 1

web media frontend

I have always wanted to tweak my HTPC frontend quite a lot to add extra functionality, but the entry barrier to learning a GUI language has been way too high for me.  I’ve had success though, in patching MythFrontend to do some things a little better for me, but I’ve always wanted to get my own going if I could.

Recently, I was thinking about how LIRC can capture IR events and map them to X keyboard events.  Basically, you can control X applications with your remote control.  I started to reason that if that were possible, then I could just use my web development skills and create a webpage frontend for my HTPC that would run on a lightweight browser, and listen for keystrokes.

Just playing around with it tonight, I actually made some really great progress thanks to a combination of a good friend, my humble jQuery beginnings, and my laughable CSS skills.  This is the result so far. :)

I’m really stoked about the implementation so far.  You’ll most likely need Firefox to get that working properly.  It will capture the arrow key presses (up, down, left, right) and use that for navigation.  I realize that the beginnings are rather crude, but the fact that I could throw this together, so quickly, while I’m just barely learning my way around jQuery seems pretty impressive to me.  I’m actually quite proud, though, that I got the navigation to work properly, too, so wrapping around rows and columns works. :)

This is certainly going to be a fun project to hack on.  If I could get this working, this would open up all kinds of possibilities for me for displaying metadata and new options for navigation.

For comparison, here’s a screenshot of what my frontend looks like right now.  As you can see, I’m trying to imitate the style as closely as possible.

There’s a lot of advantages to having it web-based — not that I’m going to serve up anything remotely or anything, this is solely for my LAN.  It’ll just allow me to build out stuff much faster.

The hard part is going to be doing testing on the frontends.  They are both running off of tiny installations, and it’s not easy building and porting software to run on them.  Sounds like a challenge that’s extremely hard, going to take a lot of time, and will have marginal benefit and at the same time increase my workload and opportunity to own more of my software stack when things go wrong.  That’s just right up my alley. :)

javascript rest api using prototype

JavaScript seems to be in my future, it looks like.  Plus a bunch of Warner Archive awesome cartoons on DVD, but I’ll get into that later.

I’ve been doing web design for a long time, but it hasn’t really gotten too interesting until I started playing with MVCs and JS frameworks and AJAX.  Now, all the *boring* stuff is totally out of the way, and the fun stuff is where I can see what I can quickly develop.  Kind of interesting really.  Plus, I think it’s kinda cool because I had pretty much sworn off getting more into computer tech, but this is one that really is interesting.  Well, for now, anyway.

So, I’m working on creating a REST API inside my MVC (using CodeIgniter), but on top of that, I’m writing a class that JavaScript can use to send calls to the REST API.  Here’s what I’ve come up with so far: link.

I’m really looking for some peer review on this thing .. the only thing you need to be familiar with is PrototypeJS and JavaScript, and it should make sense.  It doesn’t matter what REST API it sends stuff to.

I want to explain a couple of coding conventions though.

The get_rest_url() function is a protected method, which is kinda cool — I didn’t know until today you could do that with JavaScript.  If you created a REST  object named, then calling the function wouldn’t work.   You can only call it internally.  That’s why it’s in the constructor.

I should probably learn to use exceptions.  The concept of them just baffles me though, and I don’t really see the point of them when you can do things procedurally.  Whatever.  I’m holding out on being stubborn on this one.

The class supposes that your REST URL will accept the uniq identifer at the end.

The AJAX requests need to be synchronous, otherwise I couldn’t access the responses.  If someone knows a way to do both, I’d be glad to hear about it … synchronous and then using bind(this) on the function is the only option I’ve found.

That’s all I can think of.  Like I said, JS is not my strongest coding skill, so if anyone could do a once-over, it’d be nicely appreciated.

For what it’s worth, the code works just fine for my needs though. :)

lost on mvc basics: where to order data

I’m really struggling recently with understanding some basics of MVC.  I’m sold on the concept, but confused when it comes to asking hard questions like which kind of data manipulation goes into which box: model or controller.  Hopefully someone can help me out, because searching isn’t yielding much.  For background, I’m using PHP, CodeIgniter and PostgreSQL as my database.  That really shouldn’t matter much, but it helps to know what the Model is accessing (using SQL).

The basic question I have is, if you are running a complex query where you would have specific instructions on how to order the data, who does the ordering? the Model or the Controller?

If it’s the Model … then how do you pass in the order by variables into a function?  Do you have an argument that takes an ORDER BY variable that is inserted into your SQL statement?  Do you write specific functions in the Model that spit out the data in the possible orders that are available from the View?

If it’s the Controller … then how the heck do you order it?  Do you grab all the data, and then run ways to sort the arrays?  That seems extremely tedious.

I could go on ranting more, but I think I’ll leave my question as small as that for now.  Frankly, I’m stumped.

My situation I’m in right now is that I’m building reports which pull together data from multiple sources, and then need to organize them based on the most simplest ordering scheme which spans multiple columns.  That is, order by this first, then that, then the third column.  That’s all fine and good, but what about when the client wants to rearrange the data?  That’s where I get lost.

And I’m not looking for some out of the box solution like using JavaScript to rearrange data in the view dynamically.  I’m not interested in something like that.

Basically, my question is, if you are customizing data presented to the View, based on input from the client, where does the lifting go, and how do you get it in there?  Like I said, I’m lost.

prototype form serializer gotcha

In learning JavaScript / AJAX, I’ve been using the Prototype library, and I’ve been really happy with it.  It’s quickly gotten me able to wrap my head around what the possibilities are, as well as providing some good docs with great examples.

I hit a snag with the Form.serialize function today that took me a while to figure out what was going on.  Blame it on a confusing doc, because this certainly could have been explained with more clarity.

The Prototype doc (the old one, the new one is hard to navigate and has less information) says that if you pass true to the getHash parameter, it will return “an object hash.”  A careful reading of that should be implied, because it returns a JavaScript hash, not a Prototype Hash object.  Big difference.

I hit upon the problem because I would serialize my form, then try to add more keys to it, using the set function.  That would throw errors, and I couldn’t figure out why.

So, there’s a couple of ways around it.  The simplest one I like is to immediately create it as a Prototype Hash, so you can do what you would normally do.

var h = $H($(‘form’).serialize(true));

learning javascript

At work, I’m starting on a new Intranet site for the company, and one thing I am really wanting to do is ramp up my JavaScript skills in the process so I can make a large part of it AJAX driven.  I’ve played with it a bit in the past, and with the awesome Prototype class as well, but up until now it hasn’t been anything more glamorous than hiding and displaying pockets of information.

I’ve been doing some reading up the past few days, and stumbled upon some great resources.  One really cool thing I found, that I’ll write about later since I’ve got some ebuilds in the works, is GNOME Seed.  Seed is, basically, JavaScript bindings for GTK+.  In other words, you can build GUI applications in JavaScript.  How freaking cool is that?  (For the curious, read this, this and this.)

Anyway, one part of doing my research about JavaScript is building a little toolkit of common functions that I can use as a fallback when I need to understand what’s going on.  I’m really familiar with PHP, so I’ve been using that as  a sort of inspiration.

I’ve got a little function I wrote, called var_dump(), which basically does the same thing as the PHP function, and with similar syntax output.  The code is here.  If any JavaScript gurus wants to look at it and give me some pointers, let me know.  I’d appreciate it. :)  Mine is making a call to Seed, but obviously it’d be easy enough to put into a browser.

learning c++

I have finally gotten off my butt and started working on a goal that I have had for a very long time: learn some C++.

I’ve got two great books (Sams, Deitel) on the subject and between them both they are easy and explanative enough for me.

I’ve just gotten over the boring stuff, like functions and expressions. I remember I actually took an Into to OOP at UVSC about 4 years ago, and I couldn’t even wrap my brain around the concept of a function.  That and the class moved so fast for me I had to drop it.  It scarred me for life and I went on to learn ASP next.  Fortunately I didn’t get too far with that, and I’ve been coding in PHP since then.  Thank goodness the C++ syntax is really similar.  That’s gonna help.

I’m actually really excited to finally learn a really powerful programming language.  I’ve wanted to for a long time, and hopefully before too long I’ll be able to help out on some bugs on more packages.  That’d be awesome.

It’s like President Faust said at General Conference one year, “the difference between wanting to help people and being able to is education.”