Today’s adventure in coding was to continue to moving Blu-ray support into my DVD ripper software. The next step was getting the priority audio codecs that I wanted when remuxing the exported tracks.
Blu-rays support a number of advanced audio codecs, including support for the same ones on DVDs. When I’m ripping DVDs or Blu-rays, I like to choose the preferred audio codec, based on the best format.
My DVD archiving tools (dart) use PostgreSQL as the backend. I won’t go into the details of my setup because that would take a very, very long time … but if you want to see the database schema for whatever reason, here it is.
In my audio table, I keep track of the formats with a short string. Here’s the names as well, plus this is in order of preferred codecs:
lpcm – LPCM unompressed audio
truhd – Dolby Digital TruHD (Atmos)
dtshd-ma – DTS-HD Master Audio
dtshd – DTS-HD
DTS – DTS
ac3 – Dolby Digital
Normally what happens in a disc structure is that the Blu-ray will have the audio tracks indexed by the best quality as well. So you’d have the first one be dtshd-ma, and then a dts with 6 channels, and then a stereo dts, etc.
That’s not *always* the case though, and since I’m pedantic, I hunt out the examples where they are not in that order.
I’m dumping the query here, because I’m likely going to want to know what it is — I’m looking for all examples where the audio index (ix) is set to one of the higher quality ones — implying that it is coming after a lower quality one (dts, ac3).
SELECT id, track_id, ix, format, language FROM audio WHERE track_id IN (SELECT track_id FROM audio WHERE format in(‘lpcm’, ‘dtshd-ma’, ‘truhd’, ‘dtshd’) and ix > 1) ORDER BY track_id, ix;
I’m trimming the results here to show samples. Here’s one where Dolby Digital is the first English track, and Dolby TruHD is the second.
id | track_id | ix | format | langcode
62551 | 48679 | 1 | ac3 | eng
62552 | 48679 | 2 | truhd | eng
62555 | 48679 | 5 | ac3 | eng
So, the question is, when selecting the audio track, how do I get the one with the best quality — or, more specifically, the one with the codec I want when multiple are available?
Normally if I was just doing a straight grab, without wanting too much complexity, I’d simply select the first English audio track and assume that’s the correct one. A safe assumption, but this is me.
As far as a dataset goes, I’ve got I think about 100 Blu-rays, and this query I’m pulling on all the audio tracks of all the Blu-rays, and I’ve only got 14 title tracks where they are “out of order.” So, very small chance of it happening.
Anyway………. this post is all about how to do a query so that I can get those in the preferred order I want.
And here’s the answer — in your ORDER BY clause, you add one for each format, and then descend based on whether that matches or not.
Writing the query will probably make more sense:
SELECT id, track_id, ix, format, langcode FROM audio WHERE track_id = 46879 AND langcode = ‘eng’ AND active = 1 ORDER BY format = ‘lpcm’ DESC, format = ‘truhd’ DESC, format = ‘dtshd-ma’ DESC, format = ‘dtshd’ DESC, format = ‘dts’ DESC, format = ‘ac3’ DESC, ix;
Doing an order by on a match of a column will return a 1 or 0 (true or false). By default it’s going to do it ascending (ASC), which would return the trues second (1 > 0), so flip it around to DESC.
Now if I ran that same query with the track id in my first example, here’s the new result set:
id | track_id | ix | format | langcode
62552 | 48679 | 2 | truhd | eng
62551 | 48679 | 1 | ac3 | eng
62555 | 48679 | 5 | ac3 | eng
You can see that the truhd one got bumped to the top!
Here’s a second example, where even though the audio track index is already in the order I prefer, I’m going to switch around my preferences for sake of this documentation.
Here’s the original result set:
id | track_id | ix | format | langcode
60444 | 46895 | 1 | truhd | eng
60445 | 46895 | 2 | dtshd-ma | eng
60446 | 46895 | 3 | ac3 | eng
60451 | 46895 | 8 | ac3 | eng
And this time I want to prioritize the DTS-HD audio codecs over Dolby’s.
The new query, where I just change the preference order:
SELECT id, track_id, ix, format, langcode FROM audio WHERE track_id = 46895 AND langcode = ‘eng’ AND active = 1 ORDER BY format = ‘lpcm’ DESC, format = ‘dtshd-ma’ DESC, format = ‘dtshd’ DESC, format = ‘truhd’ DESC, format = ‘dts’ DESC, format = ‘ac3’ DESC, ix;
And here’s the new result set:
id | track_id | ix | format | langcode
60445 | 46895 | 2 | dtshd-ma | eng
60444 | 46895 | 1 | truhd | eng
60446 | 46895 | 3 | ac3 | eng
60451 | 46895 | 8 | ac3 | eng
There is another way to do it, which is using CASE statements inside your query. That’s what I do on DVDs where there’s only DTS or Dolby Digital. That’s another story, though.
So, now when I rip and remux my Blu-rays, I can keep only the audio tracks with preferred codecs that I like.
Oh, and if you want to see my apps that I use to copy my Blu-rays, it’s in my bluray_info github repo, the bluray_copy app, which I also wrote.
The irony of all of this is my audio receiver only supports 5.1 channels. But I can’t imagine how that’s relevant. ;) Enjoy your audio!
So, I decided to try and find a cheap little MP3 player that could also play videos that I could get for my kid brother, and instead of deciding to shop on Amazon and do the whole let’s be patient thing, I instead hit up all the retail stores until I found what was available right then. I ended up with an “RCA Wireless MP3 Player”.
I read a bit about it online looking at its specs, and it does indeed play video, but its listing of formats is non-existent. Which, while it would make most people run away, I am of course intrigued and wondering what kind of horror I can find in this thing.
The box itself doesn’t mention anything about video playback and if I’d seen it by itself in the store, I would have just ignored it to begin with. It does say “Full Color Display” on the back, though, which is still ambiguous. There’s a little video icon right next to the music one, and that would have been enough for me to grab it right there.
So I bought one, to see what it can do. It’s not unusual for the documentation or displayed features say it ONLY supports one format or codec, and then they actually have more. I’ve seen that happen with … something else … can’t remember what it was. I think maybe it was my PSP 1000 that said it couldn’t support main profile of H.264, but it did with one tweak … or something… anyway.
The docs say that it supports AMV playback — a video format that is a variation of AVI. I looked to see if ffmpeg / libav supported encoding to it natively, and they don’t. They can play them back just fine, though. So that’s good. (Note: you can get some good samples from MPlayer’s sample repo.)
Oh, maybe ffmpeg does do it. Hmm. Oh well, I couldn’t get it working anyway, but I haven’t tried *that* hard yet. I’ve got 3.4.2 here on my laptop:
$ ffmpeg -codecs | grep -i amv
DEVIL. amv AMV Video
D.A.L. adpcm_ima_amv ADPCM IMA AMV
And on my multimedia server, libav 12.2:
$ avconv -codecs | grep -i amv
D.VIL. amv AMV Video
D.A.L. adpcm_ima_amv ADPCM IMA AMV
It’s not at all unusual for the libav project to drop codecs, containers, formats, etc. that are legacy or unmaintained or whatever else, so that could explain why there’s no encoder.
(Maybe it was my old Samsung flip phone that supported a higher level of H.264. I still can’t remember. Insert note here that if you have old hardware that can playback video, I’d love to tinker with it if you wanna send it my way.)
Ran that, built it, it complained bout building the docs, but I didn’t care. The binaries compiled. Oh yes, it failed on building snow as well, so skipped that.
Encoding something using instructions I’ve found on otherblogs *runs* just fine, but the quality of my output is much lower than, based on the samples I’ve seen, could be.
The player shipped with their own AMV video, RCA_Bluetooth_Video_converted.amv which actually looks pretty good. These are the full 160×120 snapshots here:
Not too shabby overall, and it makes me want to actually get this working.
It’s worth noting at this point that I did something similar for my Sansa Fuze, getting really small videos to look really good, despite the video dimensions and hardware, so duplicating the same thing here is not really much of a stretch. Different hardware mostly.
I tried using amv-ffmpeg to do some encodes, and I could get it running just fine against a DVD source (using dvd_copy, of course). I had DSV sitting on my desk, so I started with that. It looked bad. Really bad. I thought that maybe my encoding settings were wrong and so I played with the video bitrate.
First off, though, here’s the baseline to get it encoding that I originally found:
Nothing out of the ordinary there, basic specs are the video size, 16 frames per second, audio rate at 22k, and a generic video quality setting.
I had some G.I. Joe sitting around as well because I had just bought it, and originally started looking at that as some encodes. Here’s what it came up with:
That actually looks *pretty good* right there. I mean, you’re already not going to expect much, but cartoons smooth out pretty well on smaller sizes so you can watch it and not really be too annoyed.
It wasn’t until I was doing my live action on DSV that I realized that the quality was about half of what it could be. I played around with it a bit, assuming it was just a fluke or me not using some optimal settings, but no matter what, it wouldn’t turn out very good.
Finally, getting a little frustrated, I just re-encoded the original AMV shipped with the player, assuming that it’d come out at similar quality. Nope.
I played with the video quality settings a bit, bumping it up to 8192k, but it wasn’t happening. There’s a good chance that my video quality settings aren’t done quite right, and that’s the real blocker right now.
I pretty much gave up at this point, though. I was already set back quite a bit that I’d have to have a separate (and older) ffmpeg installed *just* to encode stuff to AMV. In reality, though, I was really hoping that I could get it working, and then I’d have a good excuse to hack on ffmpeg some and get the AMV patches up to speed.
Oh, well. For my little brother I’ll get him something else that he can watch video on, but for me, I love the idea of having something that I can actually playback this interesting format that I know so little about.
At some point, I really want to detail my entire setup for organizing, cataloging, ripping, encoding, and storing my entire DVD collection, because frankly, it’s kind of cool and it’s really fun. Plus, I’ve spent a lot of time on it, and I think it’d be good to get the information out there.
Or in other words, take the braindump that is on dvds.beandog.org and put it into a huge document.
That’d be fun.
Here’s all the components (in no real order) that I have:
Command-line tools to get all the metadata from the DVD I can and drop it into a database.
Website to collect and organize DVDs and its properties (tracks, chapters) into libraries (TV Shows, Movies) and episodes or content.
Command-line scripts to extract the data from the database and create scripts to extract everything.
Do the encoding.
Every time I try to describe it, it gets really complex. I basically have to merge multiple components with multiple programs written in multiple languages to encode content to one specification to play back on one application.
It works. :)
Here’s a sample encode running right now on my server, to kind of illustrate every step I have:
I seriously don’t know how I would *organize* all the data, because of its size and complexity, and thinking about that and knowing I can go so many different directions with it and me being so pedantic about writing it, kind of makes it impossible. I guess maybe it might make sense to split preference from technology. Or something. For me, they just go hand in hand. I mean, what good is all the tech involved if you can’t get the social experience you expect (good content in good viewing situation).
It’s late, which means two things — first, I should be in bed, and second, I’m writing the coolest code evar.
With my little DVD toolkit programs I’ve been writing to learn C, I’ve had the idea bounce around in my head a lot to write a tiny (tiny, super tiny, seriously tiny) DVD player. For no other reason than to learn more C. I’m not interested in features in the least (right now), since I don’t even *use* a DVD player to debug ripping and archiving all my DVDs. Well, that’s not true. I’ll use VLC if I don’t want to patiently track down the valid track on a poisoned DVD layout. But that’s rare, and it’s just for gathering information.
There’s a couple of other ideas I’d thought about with doing direct DVD to some kind of manipulation, and each time, the idea is either shut down as in it’s stupid or it’s just completely unnecessary and something I’d never use. One was to write “dvd_mkv” which would just extract the data off a track, and dump it directly into a Matroska container. That’s a cool idea and all, but I’d never need it. In fact, I went the opposite direction and added the feature to dvd_copy to print to stdout, and I just use avconv to dump the stream straight to a MKV myself. It’s simple, and I can select the streams. Perfect.
That program’s already in my tree. I need to roll that out to distros though. That’s another post though.
So, getting raw data off isn’t a big problem (although I can’t use that method to get vobsubs off, booooooo), and there’s no real reason to hack on something.
I have tried, though, in the past, and it’s been really frustrating. I’ve tried a lot with the libav libraries, and I just can’t get it completely figured out. I’m not going to rail on their code or applications or libraries in the slightest, because that’s certainly not the case, there is *nothing* wrong on their end. There’s a high learning curve (which I’m sure few people would disagree with me), and I’m an absolute novice at C, so, realistically, the goal is pretty much out of reach at the time. Combine that with the fact I’d never use it, and I kind of meh out. (I should note though that I did get some code written so I could parse enough data on a DVD directly to get some video info. A small amount of metadata, but still, it’s cool as an accomplishment. I just noticed it’s in my tree as well. Go crazy.)
Moving on, though, if the barrier to entry was much smaller, I might be more interested in tackling it.
The libmpeg2 project is amazing, and the code is really simple to use (as in, I can read it and partly understand it), and it’s what libav* uses for decoding anyway. I managed to write a program I use as a test case, dvd_extract_mpeg2, which copies the raw video straight off of the DVD — unencrypted of course, and it filters out the garbage in there as well. I’ll probably pull it from the repo at some point or another, but for now you can see the code here. I should make it really, really clear that I *did not* write the MPEG2 decoding in there, it’s wholesale lifted from libmpeg2’s code (open source free software for the win, yo).
*That* one was a lot of fun to write, because it was the first time I was accessing the data directly and writing straight to a file. This was way before dvd_copy was a thing, so it was pretty exciting at the time. The code works, and I actually *do* use this one a lot even though it was never really intended as anything more than a fun tool to build, and it comes in handy when I just want to quickly look at the video of a source DVD. I don’t really need anything to extract the audio, because I either in the past would re-encode the AC3 audio, or do what I do now which is copy it straight to the target stream, so I never bothered writing anything. I probably never will.
I was playing with libmpeg2 for some reason this week … probably debugging some video … and I realized that mpeg2dec has an option to playback video directly. Hmm. That gave me an idea. Then I saw that it could use SDL as an output option. Hmmmmm. That gave me more ideas. I started thinking to myself, “If there was a simpler way, or at least, an alternate way to display the video, maybe I should look at that?”
So I started looking at libsdl. SDL (Simple DirectMedia Library) really is that … it’s really simple. I started reading through the documentation, and it’s super easy to understand. It’s one of those things you look through it, and ask yourself, “is that it?” thinking you missed a major part somewhere, as if there was more to it you were completely unaware of. It’s really simple. I loves it.
I’ve slowly been walking through the API documentation and poking at their samples and writing tiny little snippets to see if can understand it, and I’ve already made remarkable progress. I can’t remember when I started, but it’s only been a few days and I already figured out how to get it display a bitmap image on the screen. That may sound simple, but consider that I had *never* imagined in my life that I’d be writing anything related to a GUI since it seemed so complex and overwhelming to me, and the fact that I could even get one tiny little image displayed in a window with so little effort has completely blown me away. SDL2 also for the win, yo.
I’ve been tinkering with the code, believing that I’m understanding it right, and I got it to display one bitmap, then load another one, and switch to that. This is huge progress, by the way.
So for fun tonight, I exported the opening sequence to Teen Titans into bitmaps, and put it in a loaded sequence of my little window program. It’s pretty awesome. :D I’ve got a screenshot here, which is only a screenshot and not the sequential video, but for me it’s a little milestone marker for me. I’m having a lot of fun. (Note the incredibly short list of headers there, by the way … holy cats!)
There’s already some cool stuff I’m learning (aside from libsdl and C), which is video and image formats in general.
One of the reasons I wanted to learn C (and likewise put it off for so long) was to take something that seemed so complex and nebulous and see if I could figure it out. For me, anything written in C was just amazing. I’d pop open the code, be completely confused, understand a tiny bit, and then completely get lost again. It was just overwhelming how hard it looked, and assumed I’d never be able to really understand it.
Interestingly enough, what finally got me to learn it was having a purpose to write applications in C — namely, stuff to access DVDs. Without that, the only reason I would have looked into it would have been to play around with it a bit. The novelty quickly wears off though, especially with something so difficult.
The same effect is happening with me trying to understand audio and video though. Again, it seems like this massive *thing* that’s just *out there* that is only observable, but never able to quite understand. It’s like the X-Files of code.
I’m already learning though, that things aren’t as scary and complex as that. The big hurdle is actually getting me to process new ideas and new ways that things *can be*. I would go into more detail about what I mean there, but I can’t even put it into my words myself. All I can say is that my understanding is growing.
So, I’m having fun. Will I ever get a tiny little DVD player written that goes into my tree? Probably. Will I ever *use* it? Probably not. It’ll most likely end up to be a hobby project, something I can be proud of and play around with it when I feel like it. Those are the best kinds of programs though, the ones where I learn the most.
Final wrap-up, I want to throw a shout out to dxr3player, which also gave me the inspiration to do this … it also uses libmpeg2 and a52dec to decode the video and audio directly, something I’d never thought of doing before. The ogle DVD player got me started thinking about alternate ways to do playback as well. Thanks guys!
When you think of Gentoo, you think of bleeding edge, right? The latest, greatest and shiniest? Dag straight! Well, you can also be bleeding, as in losing blood and about to asplode if you don’t get it some first aid right away. I kind of like both versions.
I’ve used Gentoo a lot over the years, and I’ve fixed a lot of old installations. Keeping an old box up and running isn’t necessarily impossible, and I really don’t recommend keeping something longer than a year outdated. I do have one of my desktops that I kind of “snapshot” it. I’ll get it to a point where I like it … and then just leave it there.
What usually happens to trigger me into upgrading is either there’s a new package out there that I want to try and requires a lot of newer libs, or even something more dramatic, I get bored.
This is just a quick howto of how I do it though (and am doing it).
I should mention at this point that this article matches closely the title … making upgrading a box *entertaining* and *fun*. Yee.
Step one: install eix and sync the tree that way.
# emerge eix
Next, tell portage to just keep going and do whatever you have to, and don’t bother me with the little details, and just change as much as you can on your own and I might look in every once in a while if it looks like you’re changing too much:
Before jumping into world upgrades, always start with the system updates first, but do actually keep close eye on these, especially if gcc needs to be upgraded (like it did in my case). So, pause a moment, see what needs to get up:
# eix -Iuc --system
Get the really important ones out of the way (glibc, binutils, gcc, etc.). Listing the installed system ones seriously takes less than one console screen. At least look at the thing.
Once those are safely out of the way, do a quickpkg on everything that’s installed so you have fancy little tarballs for when (not if) something breaks. As in, breaks for you. I’m pro at this. I’ve broken dozens of systems before .. uh .. yeah (and fixed them, too!).
# quickpkg --include-cfg=y `eix -I# --system`
Actually, while you’re at it, make sure portage is always saving tarballs of your installed packages in make.conf:
Once you’re feeling confident that the very basics are installed so you can do whirl-o-fun, this is where it gets really interesting. Tell portage to do upgrades in a random order, but to be nice, ask you if you want to do that package or not:
# for x in `eix -Iu# | sort -R`; do emerge -uq --ask $x; done;
Then just check in on your box every few whenevers to see how it’s doing, and approve or skip updates. That’s it.
I mean for this post to be kind of stupid and it turned into being somewhat serious, so here’s something even more serious: don’t do this because this is how I do it and the last thing you want is to have to search google for “that one weird gentoo dude who told me how to break my box” because I won’t be there to save you.
To discredit me even more, I’m also that guy who will throw this in his make.conf:
So, yeah, go crazy. But remember, I’ve been breaking stuff much longer than you have … and fixing them too.
Final notes: If you’re determined to get something useful out of this post, here it is:
Always use “quickpkg” if you think you’re about to do something risky.
Having “buildpkg” in your make.conf is best practice. Everything goes in “/usr/portage/packages”
“eix -I#” — display installed packages, name only. Use –system for system, and –world for all.
If you ask for support from someone, they’re going to tell you to do a new install most likely. Use eix to get you a list of installed packages so you know what to port over.
I can speak from experience that upgrading boxes even years old is possible … depending on how much patience you have. It’s possible. I really have done it lots of times.
I’ve been thinking for a couple of weeks now that maybe it’d be good to post some stuff about multimedia on Linux again. Good for *you* that is, o snap!
Just kidding. I do have a ton of stuff floating around in my head though, that’s not all that organized, or helpful if it doesn’t have any real way to explain how to use. Case in point my awesome (awesome to me, not you, z snap!) DVDs wiki where I have a looootttt of stuff. A wiki is great for braindumps, but not for writing articles.
I’ve toyed with the idea on and off, and one of the reasons I don’t want to post stuff on my wiki is the likelihood of someone finding it pretty low. Another big reason is that I don’t want my braindump wiki to begin to have *any* sort of expectation or order or other-party direction … the site is mainly for me, and you’ll see that browsing around that a lot of it is “oh yeah, here’s the encoding settings for this random thing and a flag you won’t need to forget.” So I guess I can’t keep it all in my head.
I don’t feel really qualified to comment on the state of multimedia in Linux too much, not nearly as much as I could years ago when I was doing a lot more development in Gentoo. The reason being is that I’ve gone completely in one direction, and one only — DVDs and media centers.
I’ve had this dream for years — fifteen to be exact — of how I wanted my multimedia setup to be. It was when I first found out about MythTV and got into it. I loved the idea of it, but I hated the implementation of it. I could never get it quite the way I wanted to. But that’s okay. What really grew out of those years of experience was learning a lot about multimedia in general. That’s what I find really fascinating, is chasing the dream of how to get where I want, and all the little stops on the way … an example being of how I play with containers and codecs that I have no intention of actually using, just so I can see what it does and doesn’t have.
My happy little media center was always just slightly out of reach, and my library of TV shows and movies grew and grew over the years as well. The thing that made it finally happen was a drop in prices of hardware.
You have to realize that ten years ago, when you have 250 DVDs, and are lucky to have 250 GB of hard drives (combined), that doesn’t leave you a lot of options. And from day one, my goal has always been to have **all** of my library available in **one** place. No way that was gonna happen. So my original goal could never have been met early on simply by logistics alone (to say nothing of encoding).
I am also really, really pick about how I like things to be, and so it legit does take me years to get used to the ideas of some things. For instance, it wasn’t until about a year ago that I finally conceded that re-encoding my DVDs from MPEG2 to MPEG4 was okay. I’ve always held firm on the idea that I prefer, above all, quality as close to the original as possible — the holy grail being that it’s not reencoded *at all*. Remuxing I was always totally fine with — as long as it was a container I was fond of, but that’s a whole nother rabbit hole.
Interestingly enough, what finally convinced me that encoding was okay (going in this random direction) was that there is source material that *needs* it before it can look presentable in how I like it. Some sources are interlaced or telecined, and some need to be cropped. And that’s about it. But what really got me changing my mind and direction was … again … hardware.
I bought a new Sony TV a few years back (about three I think) that was running Android TV on it. I remember I got it for fairly cheap, brand new, at about half the retail cost everything else was going for. It’s also a 4k display. Well, long story short, I didn’t like it. It was too bright, mostly, and the idea that I had to boot up my TV and *wait for it* was beyond unacceptable. What’s the point of having a multimedia library on demand, if I couldn’t watch it when I demanded to?? (I have since realized that it only needs to boot up once, and then stays up … but hey .. first impressions are everything, and I was not impressed)
So, I would use it, and then not use it, try it, and then not try it, on and off for three years. It was too big to be a monitor, and it was smaller than my rear projection TV, and so, there was no real actual practical purpose for it.
I call situations like that “a solution without a problem.” I had it, but I didn’t need it, and when I tried to create a need for it, it wasn’t able to duplicate what I wanted, or do anything better.
The whole point of this though, is how it’s actually hardware that has changed everything for me.
Jumping back a bit to the storage problem. Hard drives just kept getting cheaper and cheaper, and I kind of woke up one day and had 1 TB of hdd space that I could use. Not bad! However, that was not nearly enough to reach my golden dream of having all my content untouched MPEG2 video and Dolby Digital audio. Nope. As my friends can attest, getting me to change my mind is incredibly difficult, if not impossible. My media library was huge — just storing Star Trek: The Next Generation was massive. To be specific, all seven seasons are 384,352 MB. That’s a third of one series just to have it on there. Hmm. Nope.
So, I had a couple of choices: encode my video (bleh), store only parts of it (bleh), or buy more hard drive space (nopes). Once again, I find myself pretty frustrated.
I also need to interject in here that it was about two years ago that I got diagnosed with OCD [Edit: I realized later that since I reference this a few times, it’d make sense to talk about it in detail, so added more info at the end of the post]. By two different professionals over a six-month period. The first time I heard it, I scoffed at the idea and said there’s no way that would apply to me. The second time, I still scoffed. But as I thought more about it over time, I realized how many ways it affected my life. Typically once I see how things *are* then I’m able to deal with them much better. I realized that it was my brain that wanted to be completely rigid, and that there were other options out there that could meet my basic requirements, even if my goal of perfection is not possible.
“If you’re watching, it’s working,” is a motto I use a lot when it comes to testing and setting up my media library. It basically means, that none of the chasing of ideals is relevant in the least if I’m not actually using the tools I’m building. It’s like building a perfect gravestone for yourself, but never considering that you’d die and be buried there. There was no practical point to it, other than to have it exactly how you want it. That’s what I was doing — perfecting my little graveyard so that everything is shiny. Media libraries are meant to be watched though, not put on display (another thing I had to learn over the years), and since I wasn’t reaching my goal, it was time to try something different. More compromises. Now that I realized I had OCD, though, it became much simpler.
Back to hardware. It played another big part when I got a four-core desktop about five years or so ago. I finally had something that could encode video at a decent rate, using x264. I think it ran at around 60 fps or so to encode video. That was awesome. That meant that I could wait about double the length of the video, and come back and it’d be done. Before this time, I’d only had dual cores, and encoding at 120 to 150 fps is kind of … a buzzkill … if you start doing the math on how long it’s going to take to encode one series.
The better hardware helped out not because I could encode my library, but because I could *test* encoding multimedia. I have spent years tinkering with everything, my idea of perfection evolving, and having intense fanatical debates in myself of why I shouldn’t use this codec, container, profile, setting, and so on (which, I realize now, is also caused by OCD). So while I couldn’t necessarily have the *library* I wanted, I could easily chase down the dream of the *standard* I wanted. And that took a really, really, really, really long time. Once again, though, it was hardware that saved the day! But not without wrecking it first.
For some reason, over all this time, it never actually occurred to me to get better, new hardware to *watch* my media. I had my trusty old Sony DVD player that could remember the playback position of six discs (SIX!!), and I loved it because one of the requirements of my perfect box was that I’d be able to resume playback of my media. Early, early on, back in the 2002s, I was already writing my own scripts for resuming playback on both a video and a series. I’d have MythTV call a wrapper script, and it’d resume perfectly. Great idea, one of my goals met, but … I wasn’t watching media. So it kind of died on the vine.
You think after reading this far that it looks like I’ve gone down the rabbit hole, believe me we have not even begun. This whole thing is a high level description.
Maybe it’d be better if I didn’t go into details, and just list out some of the things I played with:
* Hacking an open-source DLNA server so that it could resume playback
* Using the Opera store on my Blu-ray player so that I could stream video that way
* Beginning to write my own website that could stream video to my PSP
* Writing code that would detect audio breaks and commercial breaks in TV shows so that I could generate chapter points (this was actually an amazing project)
All of that is insane, but again OCD is all about being *locked* into one thing. Part of the mental circus is that there is only *one* solution. Also, that list above is not complete. But that one solution features only the hardware that I really like. Things have to be exactly the way I want, even if that means using an old Blu-ray player from five years ago that can play videos in a Matroska container with VobSub and chapters, but only if it’s plugged into a USB (I legit got an entire series encoded and put on a thumb drive just so I could watch it this way .. I didn’t).
While this may all seem like a huge waste of time, the reality is that there are huge payoffs — I learn what I want by learning what is out there, researching those paths, accepting or rejecting them. Because I’m so picky, every piece of new information I picked up on, might be the solution I was looking for, and so it was scrutinized and tested. So, I learned a lot, and that’s where all my multimedia know how has come from, either from things going the wrong direction I want, or usually more because it’s fun to tinker with and see what I can do. In fact, one of my favorite hobbies is to get a device and then in a lot of detail figure out *exactly* what video specs it contains (for example, did you know that the PS3 supports one kind of video if it’s on USB vs streaming over the network?). The point is that even if nothing was what I wanted, I knew all the reasons why. That’s knowledge, yo.
Remember how I bought that Sony TV and then never really found a use for that? That’s a common story with me — I’ll get *some* hardware because it looks fun (and it is), but ultimately end up with something I can’t really use. So I had this tiny little Zotac PC that I bought years ago, but couldn’t really do anything with. It didn’t have wireless, so I couldn’t just put it anywhere (which was an oversight on my part … when I bought it I told myself “nah, I’d never need that !), it only had one Ethernet port, so I couldn’t use it as a router, but it *did* have a sexy nvidia card on it that could do hardware decoding. And it was fanless, which made it even sexier. Still, though, a solution without a problem. The problem I had, again, was that there was no storage space … or not enough.
I’d grown up with desktops so of course all I ever had were the huge 3.5” drives. I had accumulated more than a terabyte by this point on my life (probably more like two, spread out across a lot of disks), but it didn’t really *do* me much good. Sure I could store video, but if I was going to have something that’d do video playback for me on my media center, the idea of throwing a desktop behind my TV and hearing the fan whir was not my idea of an enjoyable media experience (Ironically, part of my enjoyable experience is to turn up the volume so loud that I wouldn’t have been able to hear the moon fall from the sky, much less my case fan).
What happened though somewhere along the line (this part is a bit fuzzy, so I’m guessing a bit at this point) is I wanted to swap out my PS3 for an SSD. I had already upgraded the drive in there from the original 250 or so GB to a 750 GB. I was hardly using *any* space on it at all though .. maybe 200 max … if that. Basically, I had all this storage that I wasn’t using, and hey … why not throw that into my little Zotac box? I did that, and then it hit me … I suddenly had a lot of space on a tiny little fanless box that can sit behind my TV and not make noise or annoy me, and it had enough space on it to hold some media, *and* to make it even better … it had a sexy nvidia card on it that could do hardware decoding of MPEG4 *AND* MPEG2 … and … hmm … is this possibly the birth of my new media center? Maybe? Maybe …. ??
There was one problem. A remote control.
You see, another part of my big perfection problem is that I had to have only *one* remote that I could use. Ever. One! Just one. The end. No bargaining on that point. There was no way to control that little bugger, even though I could put media on it. This was where I came back to looking at DLNA servers a bit more closely, and started delving into what possibilities it’d hold.
I liked it, and I didn’t like it. I liked it because it had all my folders and I could easily *browse* my library, but it just wasn’t … fancy. There was no cover art, and resuming playback with my DLNA clients only worked in that powered up session. So. It was kind of a pain. It was close! And it was very close, and for the first real time I could actually get a lot of media on there. So that was a good start. I had a proof of concept, at least.
Resuming playback again had always been a big goal of mine, and I could at this point sacrifice continuing on a series, because I’d just remember where I left off, and I could just select the next one. That’s one concession I’d be willing to make, but there was also one workaround I could do as well — for the individual episodes, I could split the media files into chapters and use that as a poor man’s playback system. It was a good idea, in theory, and I probably would have pursued it further until I ran into another problem — a lot of shows on DVD don’t have chapter markers … which means that I’d have to go through them and find them myself. What about if I just cut it in five minutes increments though? No. Maybe the show has the same commercial breaks in every episode. Nope. Maybe, test, maybe, test, maybe. Nope, nope, nope. Nothing was working out.
Oh, well. I hadn’t had a solution that really met my needs in a long time, so I wasn’t really disappointed per se. I had just found another solution that didn’t fix my problem. It was fun to play with, though, and I did a lot of documentation on what would work streaming vs. local playback (another fun intricacy, my Blu-ray player would support different formats + codecs over DLNA than it would on USB .. same with the PS3).
Blah de blah de blah. If you’re getting bored reading by this point, coincidentally I’m getting bored by writing it, so I’ll wrap it up.
At some point, I discovered Plex. I don’t remember all of the details about how I found it or got started with it, and I think it’s because it all happened *so* fast. It probably overnight became the solution I was looking for. Here were some of the features that just slapped me in the face:
* Automatically fetches metadata
* I can *override* the metadata and use mine if I wanted
* PS3 app
* Web frontend that was sexy-go-nice
* Most importantly, direct hardware playback … no encoding!
Since I had played with the PS3 and (by this time) my PS4 with DLNA so much by now, I had really warmed up to the idea of using my game controller as a second remote. It didn’t bother me. Plus I’d use it to stream Netflix and Amazon Video since I liked it better than my Blu-ray player, and it worked out well. So, the timing was perfect and everything just kind of fell into place. And that’s what I’ve been using since.
I’ll go into more details in other posts about *how* I use Plex and *how* I encode stuff so I get direct hardware playback and all that stuff, but this post is just all about the journey.
Suffice it to say, that after fifteen years, I finally got the media setup I wanted. And I’ll explain how.
I have meds now that help me not super-hyper-focus to a point where it’s detrimental in my daily life.
Being labeled with a mental illness for me “gives a name to the beast,” and by so doing, it helps me realize the source behind some madness. That’s what happened here. I still “have it,” but have much better perspective knowing it’s there, and am able to circumvent it by acknowledging it’s presence and “input” on ideas.
OCD stands for Obsessive Compulsive Disorder — however, did you know that those are two different disorders lumped into one? It’s not unusual for that to have a diagnosis cover a spectrum of issues. There’s the obsessive disorder, and there’s a compulsive disorder. Someone may have one, the other, or both. In my case, I have the first. I get obsessed with how things are in my life, the most common being trying to reach a level of perfection — this entire post illustrates that point. It doesn’t affect me too much other than that, since it’s highly personalized on what I’m obsessive about, which is almost always, the way things are set up for things that I highly care about. Meaning, not everything.
It’s common when someone hears about OCD to picture a person scrubbing their hands multiple times when washing their hands when they bleed. That’s compulsive. I don’t have that, thank goodness, and I can only imagine how stressful for people it would be that do.
I have a somewhat annual tradition I like to do around Thanksgiving, which is go visit random friends and family that I know, drop off a pie, and at the same time pick up any spare clothes that they may have, and then take them to the homeless shelter. I’ve never really been a big fan of Thanksgiving, since I’m kinda meh about turkey, and I don’t really like sitting around all day. Thanksgiving, however, is a perfect holiday when everyone else is home, so it’s easy to stop by almost any place and pick something up.
I’m going to try and do this again this year, but this time around be a bit more organized. So … here’s specifically how it works.
I make lots of chocolate cream pies, and if you live in either Salt Lake County or Utah County or Davis County, I’ll swing by when it works for you, deliver one, and ask if you have any spare clothes or items you can donate to the homeless in Utah. I’ll be out and about all day WEDNESDAY Nov. 26th through SUNDAY Nov. 30th, and can come any time if there’s a preference.
What you can donate: clothes, obviously, are really appreciated, but here are a list of things that *I* myself am always surprised to hear what they need:
As far as clothes go, we need them of ALL sizes (including big and tall, and plus sizes) and for ALL ages — yes this means babies, small children and teenagers along with adults. The biggest ones out of anything need to be UNDERWEAR and SOCKS — again, for all ages and sizes. One time a volunteer at a homeless shelter told me “they wear them until they literally disintegrate and fall apart.”
If you can donate something, great. If you *don’t* have anything at your house to give, please go to the store and spend $10 on any of the above items. It’s small amount to spend, but it’ll make a huge difference!
I haven’t decided yet which place specifically I’ll be taking the donations to, but chances are I’ll spread it out based on what I can pick up. Here’s places I’ve taken stuff to / helped out at before:
The Road Home – an organization that helps homeless transition back into jobs, housing
Palmer Court – The Road Home’s permanent housing for homeless who need it
So, I started learning C lately, and my life satisfaction has gone way up. In retrospect, I started thinking about all the things within the past three years that I’ve started doing — and they are all things that I’ve wanted to do since I was like 12 — and how much absolutely fun life has gotten since then.
learning how to play and be a Dungeon Master for Advanced Dungeons & Dragons (2nd edition baby, yah! nothing but!)
reading Star Trek: The Next Generation novels
switching to paper plates
(I could go on for a while .. )
If I go back a few years before that, then we have me going back to school and studying psychology, which has also been a major life event. And going back a few more than that (so we’re at about 8 now), it would be getting involved doing community service.
So, if I had some words of wisdom to pass down to a lot of people at once, this is what I would say: you can do all your grocery shopping in the frozen foods aisle.
Working on C has been a huge amount of fun for me. When it comes to working with computers and/or development, I love doing things the absolutely most complex and difficult way possible (a la Gentoo). Why? It’s because if there’s some knob I can turn, or some option that I’m given, I want to customize it. Originally it used to be because … well, I dunno why … but I’ve definitely mellowed out a lot over the years. I’ve always dreamed of learning C, though, mostly, because it’s just *hard*.
I mean, the language only has a few data types, and it’s super easy to screw up your memory, your life, your grocery store, you name it. Things just got real. Which means things just got fun.
At the risk of sounding like a snotty kid who wants everyone to know how awesome he is, it occurred to me late last year, that of all the things computer-wise I know about a lot, it’s probably DVDs. Having a moderately-sized library (a few hundred or so … or a few hundred more than that few hundred …), I’ve ripped and accessed all of them, and because of that, I’ve found lots of tweaks and bugs and ways to get certain information out. It kind of clicked with me that, I had a huge wealth of knowledge in my head about one subject, and it may just possibly be something someone else could use. Since I doubt anyone would give me an honorary Ph.D., I had to do the next best thing — slap together a half-finished wiki of my braindumps on the matter.
Honestly, I’m doing it more in a sense of the posterity of knowledge than anything else. Going through my notes, I have a lot of stuff that I just dumped on there. There’s really not *much* content there, but what it does carry behind it is years of working with DVDs on Linux … so when I say “use Handbrake,” I have months of research of saying why that is, or something similar. Like I said, I’m a user and I’ve run into every situation imaginable, so it’s good to be able to kind of share that. And let everyone know how detailed I can get. Because people love that kind of thing. Yahp.
However, learning C has taken me to the next level, where I can do two things: apply patches to the main DVD libraries, and write my own C programs. I would be the *last* person to claim that this stuff is elegant code or wonderful programs, but it does, for me, have the goal of driving towards the point of providing data that users need. Whatever that phrase means. I think I’m trying to say that I like writing stuff that is for *intermediate* users. I used to work on the Gentoo wiki a lot, and I noticed I really liked writing documents that go beyond a basic “this is this and that is that” explanation, and instead dives into the perspective of “let’s assume you know the basics, and want to do something that’s a little more detailed.” Yah. That.
I don’t know where I was going with all this, but there’s two things I’m sure of. No, one. I am out of frozen pizza.
It has been a looong time since I both posted in my blog and worked on my custom DVD ripper scripts. Apparently the last time I worked on the code was last June, and even then I didn’t make many updates.
I’ve been spurred onto building up my DVD library again by a couple of things. First, I realized that my Blu-ray player has support for Matroska videos with VobSub and SRT subtitle support! I was not expecting that. In fact, it’s way better than what my PS3 can playback, which is … depressing.
I put away my HTPC about two years ago, when I was living in my previous apartment. I moved into a place that was probably about 550 square feet. Pretty tiny, and I liked it, but no room for a fantabulous multimedia setup. So I sacked it for a while and was okay with that. The fact is I actually spent more time getting it up and running and customizing it than using it. Which is weird. Actually I spent even *more* time ripping the DVDs and then not watching them. But that’s okay. It wasn’t until recently that I found a setup I think I’d like even more.
For now, I’m preferring having *less* hardware, and so just sticking a small 8GB USB thumb drive in my Blu-ray player with a smattering of samples of shows suits me just fine. It’s no amazing thundershow of hardware and multimedia, but it *does* get me actually watching the content, so there. I imagine if (and when) I have a house where I can properly get loud without upsetting neighbors, that’s when I’ll whip the big speakers back out and deck it out properly. Some day. :)
In the meantime, today, I’ve been working on my DVD scripts. I call it dart for “dvd archiving tool.” It’s a complex set of scripts that I’ve been putting together for years, and it is highly customized for my own setup, with a CLI tool to read and access DVDs, then archive them in a database. I also have a web frontend that I use to tag tracks, titles, episodes, etc. and so on. If it wasn’t so unwieldly I’d throw the source out there, but the thought of having to explain to *anyone* how to get it up and running makes my head hurt. So, if you want a good DVD ripper, here’s my advice: use Handbrake.
One problem I was trying to solve tonight was checking for these three statuses of my DVD drive: is the tray open, is the tray closed, is there media in the tray (while closed). I have to use different tools for each one, but the problem that I always run into is this: it’s impossible (as far as I have been able to discover) to know when a DVD tray is both closed and ready to access.
The problem is that you can run eject just fine to close the tray, but once the command exits successfully, that doesn’t mean the drive can be accessed. That is, running “eject -t /dev/dvd” and then “mplayer dvd://” in sequence, mplayer will complain that there’s no DVD device.
What’s the solution to all this? Well, wait four seconds after running “eject.” That’s simple, but I still spent hours today trying to find out if there was another way to do it. While I never did (and ended up using ‘sleep’), I did find some cool stuff for polling and reading DVD devices.
blockdev basically displays some interesting information about the block devices — in this case, /dev/dvd. Now, for my library, one thing I have been doing lately is storing the size of the DVD in my database, so I can get an accurate number of how much HDD space I need when I want to archive the UDF or rip it.
You can use blockdev to get the amount of bytes like this:
blockdev –getsize64 /dev/dvd
Now if you want to see that in megabytes, just divide it by 1024
expr `blockdev –getsize64 /dev/dvd` / 1024
Next up is udisks, which can get information about the DVD device itself. In this instance, I use it to see if there is media (a DVD) in the tray or not.
Running “udisks –show-info /dev/dvd” spits out all kinds of interesting information, but what I’m looking for is the “has media” field.
udisks –show-info /dev/dvd | grep “has media”
has media: 1 (detected at Wed Jul 3 23:21:23 2013)
Now, that will say 1 *if* the both the disc tray is closed and there is something in there. And if the DVD drive has stopped spinning enough for the command to work (again, sleep 4 seconds after closing the tray).
It will display a zero if there is no media *or* if the DVD tray is open. Here’s a simple command to get just the number:
This is an old small command-line tool I’ve used in the past. It polls the drive to see if there’s something in there or not, and if the tray is open or not. Sounds great, right? It should do everything I want, solving all my problems … except that it doesn’t build on my system (Ubuntu 12.10 with gcc 4.7.2). It used to, on my older setup, which would have been about 2.5 years ago.
It’s just a small C script, just over 500 lines, you can find it here on Freshmeat. If someone wants to patch it to get it working, I’ll personally deliver you a plate of brownies. Mmmm, brownies.
I actually *do* have an old 64-bit binary that I built way back when, because I kept a copy of my old development filesystem. So I have a working blob, but it kind of breaks. So I kind of rely on it. I can only use it if the tray is open or if the tray is closed and empty. So the way I check if a device is empty and closed in my script is I’ll first poll it to see if it has media with udisks, and if it doesn’t, then I’ll run this one. If I run it with a disc in there, it pukes on me, and so I have to work around it. It’s a hack, I know, but whatever.
qpxtool and readdvd
This is the project I ran into today, and I am super, super excited about it. The QpxTool project is full of way cool little utilities for accessing your drive settings. Honesty, I didn’t look at the other ones, because I was so hyperfocused on ‘readdvd’.
From the man page, “readdvd reads even a corrupted dvd and writes the the result into a new image file on your harddisk.” This is awesome, because it’s the first utility I’ve found *specifically* for creating an exact image of a DVD filesystsem (UDF). In the past, I’ve always used dd, but now I’m onto this one. It skips over bad sectors and gets the image squeaky clean off of there, and I could not be happier. This one ranks up there with Handbrake in both awesomeness and must-have-ness. I should add that it’s also in Ubuntu’s default repos, so have fun.
Just run “readdvd -o movie.iso /dev/dvd”. Pretty simple.
That’s pretty much it for now. There are other great tools out there: lsdvd also ranks in the “must have” category. I couldn’t do anything without it.
I mentioned dd earlier, and I actually use pv with it to give me a nice progress bar (also in Ubuntu repos). It works just fine, I’ve been using this approach for years.
pv -ptre /dev/dvd | dd of=movie.iso
One more thing I wanted to mention. Sometimes, some errors get thrown to the syslog because either an application or the DVD drive itself is being fussy. I haven’t quite narrowed down which it is, but I’m betting it’s the firmware on the DVD drive complaining since some brands (Memorex) complain, and some do not (BenQ). By far, the best-quality DVD drive I’ve had to date was actually a Sony BD-ROM drive. At least, I think it was Sony. Here’s some of the errors I get sometimes:
To avoid issues like this, I run a small command to just decrypt the CSS on the DVD so it can kind of clear its head a bit. Just run mplayer on it, watching about 60 frames (or 2 seconds worth of video), but just ignore it and dump it out. The whole point of it is to decrypt the DVD, and move on with your life. And here you are:
I don’t pretend to understand how or why that helps, but I know it does. If someone knows why the drives are doing that, I’d love to know.
The only other app I can think of right now off the top of my head is ‘dvdxchap’, which is part of ‘ogmtools’. I know ogmtools is old, and the OGM container isn’t popular anyway (that I’ve seen), but it’s perfect for getting the chapter information out. Although I may use something else now (lsdvd?). I can’t remember, and I haven’t had to mess with chapters lately.
That’s it for me. Have fun, rip away, and watch some cool Super Friends DVDs. There are a LOT of seasons out there. It’s great. :)
Every year, UVU (my school) hosts what’s called The Clothesline Project. It’s a project that is designed to raise awareness about domestic abuse of all kinds — physical, emotional, sexual, and also things leading to death and suicide.
The layout is that survivors of abuse create t-shirts where they tell their story or share a message about what happened or what they went through. In some cases, the t-shirts are written by relatives of people who were abused and were killed. It’s pretty intense stuff.
I went last to last year’s exhibit, and went again this year for my psychology class. The first time I went, I looked at almost every t-shirt there, and I was at the exhibit for probably an hour and a half. On multiple occasions I was so grossed out that I almost threw up. A lot of them were terribly traumatizing. I remember driving home afterwards, and I was so overcome by emotion that I was sobbing uncontrollably. Afterwards, I was deeply depressed for about a week. It certainly raised some awareness in me.
I learn a lot from reading first-hand accounts of anything related to situations like this. It gives a clear, non-academic approach of what it feels like for someone to go through these things. The stories are informative, to see how they cope, how they escape, how some of them let go, and so on. There are all kinds of endings as well. Sometimes their family or friends don’t believe them, sometimes the perpetrator dies or gets thrown in jail and is convicted. Other times they get a divorce, or get married to someone else, or just flee the situation completely.
A common problem that I see in a lot of the stories are this — people do not speak up when they are being abused, or do nothing about it. In some cases, someone else in their family was also being abused, but neither one knew. It is so important to speak up, to tell someone! Abuse has many side effects on the person receiving it. It severely mess up their emotions and take away from them a proper healthy reference of how things like relationships, sex, and emotions are supposed to be. The best comment comes from one of the shirts below: “Silence is your enemy. Talking is your medicine.”
I took some snapshots with my phone this year, because I wanted to post some of the stories on here. I only managed to get a few, because I showed up at the display when there was only about twenty minutes before closing. On top of that, I opted to only take pictures of shirts that I thought I’d be able to read later from a photo.
I’m posting the pictures and the text some of the t-shirts on here. Be warned that these are graphic, verbose, and terrifying. Proceed with caution.