All posts by Cory

Hugs - Painting by Cory Gugler

Stuff I found Interesting (June 2014)

An interesting article about ‘The Scroll Up Bar‘, a navigation trend on the web that has recently been becoming more popular. I’m a fan of this style as it really helps save some valuable screen space when browsing on devices.

Defensive BASH Programming – A load of techniques for BASH programming. I picked up a few good tips for cleaning up some BASH scripts I’ve written.

One Top Quest – A surprisingly entertaining web game considering how simple it is.

JSNice – A neat JavaScript deobfuscatator that uses statistical renaming of variables and attempts to guess type annotations. From the few examples I tried it seemed to do a decent job of making obfuscated code readable.

Human JavaScript – free online book about building browser apps.

Liftoff painting by Cory Gugler

Resizing a VirtualBox Virtual Machine (Debian) hard disk with LVM

Update October 2018: This article is pretty old and I now generally use docker for development. Proceed at your own risk!

Recently I was backing up a site to my development VM when it suddenly started complaining about lacking available space.  I started looking around in virtual box to see where I could increase the disk space only to find out it was not going to be a simple task.

I found quite a few good resources for resizing a VM hard disk, but none fit my setup completely, so I thought I’d document it here for future reference.  Of course, make sure to backup data before you start working on resizing your own VM.  I’m currently running VirtualBox 4.3.10, your setup may differ from mine.

Remove Machine

First step is to remove the VM from the VirtualBox Manager.
Select: Machine -> Remove.

Dialog Box - Remove Machine

Make sure to click ‘Remove Only’, else you’ll have nothing left to resize.

Clone & Resize the Disk

Open up a terminal, navigate to your VM disk, and clone the existing disk with the command line tool vboxmanage.

vboxmanage clonehd ./Old_VM.vdi ./New_VM.vdi

Terminal command to clone disk

This may take a minute. While you’re waiting, go ahead and download a GParted Live Bootable Image. We’ll use this to expand the partition once we’ve resized the disk.

Before we resize the cloned disk, we may have to remove it from the VirtualBox Manager. In the VirtualBox Manager GUI, navigate to File -> Virtual Media Manager and remove the new disk.  Again, make sure you keep the file.

Now we can resize the cloned disk with vboxmanage.

vboxmanage modifyhd --resize 20000 ./New_VM.vdi

Resize the Partition

Now we can add the machine back into VirtualBox. Before we can add it though, once again we need to go to File -> Virtual Media Manager and remove both the old disk and the newly resized disk.

Dialog to remove disks from VM

Now add the machine back under Machine -> Add. You’ll need to navigate to where your virtual machines are stored and select the correct vbox file.

Once the machine is back in, open the settings on it and swap the disk drive in the storage section.  This is where you will also mount your GParted iso into the VM, so when we first boot up we can resize the partition.

Dialog to change disks in VirtualBox

Run the VM and when GParted boots, resize the extended partition and then the lvm partition and reboot the VM.

The VM should have ejected the GParted disk and now you’re VM will load as normally. You’ll need to make sure you have both the lvm2 package and resize2fs (e2fsprogs package in Debian) tools installed.

We can use pvdisplay (run as root) to show info about the logical volume, including it’s path and size. Resize it to the desired size using the LV path to specify the volume:

sudo lvresize -L +4G /dev/addlimedev/root

Now we just need to use resize2fs to expand the size of the filesystem.

sudo resize2fs /dev/addlimedev/root

And verify we now have our newly acquired space:

df -h

Terminal Resize Filesystem

Success!  A little more involved of a process than I’d like, but that’s all there is to it.

Resources

Here are a few helpful resources I came across while figuring this whole resizing deal out.

Resize LVM PV Partition in Debian/Ubuntu
Extending the size of a virtual disk in VirtualBox + resizing Linux LVM partitions
forums.virtualbox.org How to resize a Virtual Drive

 

Shatter.js Image Shattering

Shatter.js – Make things explode in JavaScript

Sometimes I feel the need to break things.  We all get that feeling sometimes after a rough day at work, or after a day of dealing with pushy clients, or maybe your cat threw up on your face while you were taking a nap. The point is, we all need to release some steam and take out our frustrations sometimes.

Now, you could pick up the flower vase sitting by your computer-you know, the one that’s been sitting there with the dried out remains of a dead plant in it for the last 2 years-and toss it across the room. But then you have to deal with little shards of glass all over your floor and you’ll have to explain to your roommate that you did not, in fact, just have a psychotic episode.

Aside from that, we now live in a digital world. I imagine most people wake up in the morning and their first instinct is to go straight to the bathroom…and take a picture of themselves in the mirror to post to facetwitgram before they post their daily half eaten breakfast photo.

So what better way than to break stuff digitally? And with today’s hottest new technology, JavaScript!

Shatter.js is a small little library that can help you do just that! Use your JavaScript programming skills to add and image to the dom, give it to Shatter.js and it will return to you your image all nice and shattered and packaged up for you to do with it what you will.

“That sound’s incredible! But, wait, how much is this going to cost me?”, you say.

Well, that’s a great question, and my answer to that is that it’s completely open source and free! Do with it what you will! It’s licensed under the MIT license and available on Github for you to download, fork, branch, pull, push, rm -rf, or whatever you desire!

Check it out on GitHub or view the examples on the project page.

Groovel - Painting

What I found interesting this week (May 18th 2014)

WTF? Forms – A nice bit of css for some common form controls.

Digging, Debugging – This talk is loaded with great tips on debugging with Chrome devtools.

CCNA Training Videos – 88 videos covering CCNA training, crowd funded through Kickstarter and now available to stream free on YouTube.

Littlewargame – HTML5 rts game, amazing performance for an HTML5 game.

And there’s this: Cats and Dogs dressed as people, 100 years ago.

Water color painting by Cory Gugler - Energist

How I tricked myself into being productive

It was Friday night, I just wanted to relax, but I have a 11 day streak going on Github. I think to myself, it doesn’t matter, I can take a day off.

I look at the bugs/feature requests I logged in github for Shatter.js, and realize, well, upgrading Phaser won’t take too long, and at least I’ll have done something productive on it today.

So there I am, upgrading the code and fixing the example. Something I’ve needed to do for awhile, and I got it done when otherwise I was going to settle down with a glass of whiskey and let my brain have the night off. I successfully tricked myself into being productive.

Sometimes just getting started is all I need to get motivated. I know once I get started on something that I’ll work longer than I intend to and I’ll enjoy it. The hard part is just getting started.

The procrastination beast has it’s hold on me all the time, keeping me from starting anything. I think it will take too much time, or I find a million other little things to do with my time and all of the sudden it’s way too late to make any progress on my project, so I’ll do it tomorrow. In fact, tomorrow I’ll work twice as long! Yeah, that’ll happen for sure!

I don’t know what my apprehension to get started is. I feel like my brain knows it’s not going to get instant gratification so it tells me I need to read a couple more articles, or browse for more things I can add to my read-later or do-later list. I often look at all the things I want to do and get overwhelmed, and do nothing at all.

Since I started trying to keep up a streak on Github, I find myself working a lot more, and a lot more consistently.

I think to myself, well, let me just get one or two commits out of the way so I can relax the rest of the night. Maybe tonight I’ll just update some documentation, I’ve been needing to do that for awhile. I start to update docs, and next thing I notice a couple hours have passed and I’ve re-factored code and added some functions that I’ve been meaning to add for weeks.

I fool myself. I know once I get started I’ll enjoy what I’m doing, but it never fails, every time I think about getting started there is this apprehension, something holding me back. But I’ve almost never come out of it afterwards unhappy that I did it.

Occasionally I’ll have some really unproductive coding sessions. Maybe I only end up updating some documentation, but at least I showed up. The more often I manage to show up the more often I’m likely to show up again.

After a few weeks I realize I’ve almost completed this project. It’s not nearly so enormous and insurmountable when I tell myself I’ll just make a few commits a day. What’s really perplexing is that I know this already. I know iterative progress is how to get things done. I’ve known all these tricks to get yourself started. Like the 5 minute rule. Tell yourself you’ll do something for 5 minutes and you end up going a lot longer once you get started. I’ve read tons of ‘productivity hack’ articles and books. But the problem comes when I know that I likely will go longer that 5 minutes, and decide to put off the 5 minutes b/c of it. It’s like double procrastination. My procrastinating brain to too clever for these parlor tricks. I know you can’t expect to build something of high quality quickly, yet I still expect to be able to pump out a bunch of great projects, but at the end of the day/week/month/year, I realize, I barely did shit. I was too overwhelmed with the scale of the projects I want to complete, and the amount of knowledge I need to learn to get there.

I’ll just take things one commit at a time.

 

Groovel - Painting

What I found interesting this week (May 11th, 2014)

Interesting things I read/watched/listened to on the web this week.

Everything is about execution. Dreams and visions are free. Implementation takes place in the real world where friction and inefficiencies exist.

A great answer on quora on life lessons.

While playing guitar I remembered an old site I learned quite a bit from: blackbeltguitar.com. Unfortunately it’s no longer around anymore. However, it’s still available on archive.og.

Found devdocs.io – A great well organized and easy to navigate reference guide for web api’s. Much easier to find and browse information than most of the official specs.

Watched this very funny animated short: Johnny Express

Feast your eyes on a live hd view of the Earth from the ISS.

Scripting visual effects for video with BASH, FFMPEG, Script-FU & GIMP filters

Recently I’ve been working on an animated music video for Mammoth Life. The concept involves creating a wax-crayon hand drawn look while mixing live action video with 2d animation. For the live action footage I needed to find a way to process it to make it appear as if it were hand drawn with crayons. Using a combination of FFMPEG and GIMP and some BASH scripting I’ve been able to come up with a pretty solid effects pipeline. I’ll take you through my process so you can process your video’s through GIMP too.

You may need at least a little familiarity with the command line, FFMPEG and GIMP scripting to follow along.

First, I’ll cut the video down to a decent length for this demo. I’ll be using the public domain cinematic masterpiece known as ‘Santa Clause Conquers The Martians’, available here:

https://archive.org/details/santa_claus_conquers_the_martians_ipod

I’ve chosen to extract a particularly tense moment in the movie from 1:09:48 – 1:10:22. We can do that with FFMPEG like so:

ffmpeg -ss 01:09:53 -t 00:00:29 -i santa_claus_conquers_the_martians.mp4 -vcodec copy -acodec copy santa_hostage.mp4

What we’re doing here is saying starting at 1:09:48, we take 29 seconds and copy to the new file ‘santa_hostage.mp4’ without reencoding.

Next, we’ll pull out a frame from the video so we can figure out what filters we need to run on the video frames to get the effect we want. I want one frame from 2 seconds after the start of the video and my source video is 640×480, so we do something like this:

ffmpeg -ss 2 -i santa_hostage.mp4 -t 1 -s 640x480 -f image2 single.jpg
Screenshot of movie
Images of Martians, direct from the Mars Curiosity Rover

I’ve already got a workflow for the crayon effect, so I want to try a new one with this video. There’s a cool effect in G’MIC (available as a plug-in for GIMP here: http://gmic.sourceforge.net/) called ‘Graphic novel’. Applying it straight to the image gives us a cool effect, but it looks a little rough due to the blockiness of the video. To remedy this I applied a 6px Gaussian blur first before applying the G’MIC effect. I think it improved it enough for our purposes.

Before and After
Half of a Martian shirt. Before and after ironing.

We could go crazy and layer other textures on or run it through more filters, but I like this as it is, so let’s get to scripting.

Processed Image
Martians – The Graphic Novel

Before we start let’s define exactly what we want to do:

  1. Extract all the frames out of the video
  2. Apply a Gaussian blur of 6 pixels followed by the Graphic novel preset on the G’MIC plug-in.
  3. Compile all the frames back into a video
  4. Add audio from original video back into new video

I’m going to start small by only extracting a few frames to start and only take 6 frames per a second so I can also achieve a choppy/jerky effect. Once the GIMP script is finished we’ll be able to go back and process the whole video.

ffmpeg -i santa_hostage.mp4 -r 6 -t 1 -s 640x480 image%3d.png

The -r 6 indicates we only want 6 frames per second and image%3d.png means the images will be spit out numbered from 001 – 006.

Now we’re ready to write a script to apply the effects, but first we need to know how to call the effects in the script. We can use the Procedure Browser ( Help->Procedure Browser ) and the Script-FU console (Filters->Script-Fu->Console) to manually enter them to make sure we get all the parameters correct.

After beginning to type in Gaussian we see all the different Gaussian blur plugins/options, plug-in-gauss-rle2 is the one I’ll use. On the right all the parameters are listed that we need in order to run the plug-in.

Program Parameters

As you’ll see in the console, GIMP uses TinyScheme, so we can run the Gaussian blur plug-in on our image like this:

(plug-in-gauss-rle2 1 9 (car (gimp-image-get-active-layer 9)) 6 6)

The first parameter is 1 for RUN-NONINTERACTIVE, which means to run without presenting the plug-in’s dialog box. The second parameter (9) is the image. We only need this number when we’re testing in the console. It’s displayed in the title bar.

Title Bar shows Image ID
Title bar showing the image ID

The third parameter is for the drawable. We’ll grab the active layer for that. (gimp-image-get-active-layer 9) returns the active layer on the image, but it returns it as a list, so we use (car ) to return the first (and only) item in the list.

The last two parameters are the horizontal and vertical radius (in pixels) of the Gaussian blur.

If you run that through the Script-FU console you’ll see the image get blurred. Success! Next, we just need to determine the command for the G’MIC plugin.

G’MIC is not just a GIMP plugin, it’s a full featured framework that can be used for many image processing tasks. In the procedure browser plug-in-gmic takes the image, drawable, a layers mode, and a command to run on the image with G’MIC.

I have no idea what command the graphic novel preset runs, but after a little research it turns out it’s really easy to get the command by running gimp directly from the console. As described here, if we run gimp from the console and turn on verbose output in the plugin we’ll get our command.

./apply/ -v -99 -gimp_graphic_novelfxl 0,26.6667,9.74641,11.634,20,0,0.62,14,0,1,0.5,0.78,1.92,0,0,0,1,1,1,0.5,0.8,1.28

So our final command in the console is this:

(plug-in-gmic 1 9 (car (gimp-image-get-active-layer 9)) 1 "-gimp_graphic_novelfxl 0,26.6667,9.74641,11.634,20,0,0.62,14,0,1,0.5,0.78,1.92,0,0,0,1,1,1,0.5,0.8,1.28")

Running that command in the Script-Fu console works, so now we just need to throw both these commands into a GIMP batch script. Let’s copy some boilerplate code from from the tutorial at gimp.org. You can see below, I’ve grabbed the last script on the page and modified it to run the two commands we need to get the effect we want.

(define (batch-graphic-novelfy blurnum)
  (let* ((filelist (cadr (file-glob pattern 1))))
    (while (not (null? filelist))
      (let* ((filename (car filelist))
        (image (car (gimp-file-load RUN-NONINTERACTIVE
                                    filename filename)))
        (drawable (car (gimp-image-get-active-layer image))))
        (plug-in-gauss-rle2 RUN-NONINTERACTIVE image 
                            drawable blurnum blurnum)
        (plug-in-gmic RUN-NONINTERACTIVE image drawable 1 
          "-gimp_graphic_novelfxl 0,26.6667,9.74641,11.634,20,0,0.62,14,0,1,0.5,0.78,1.92,0,0,0,1,1,1,0.5,0.8,1.28")
        (gimp-file-save RUN-NONINTERACTIVE
                        image drawable filename filename)
        (gimp-image-delete image))
      (set! filelist (cdr filelist)))))

I named the script ‘batch-graphic-novelfy’ and added the two commands we want it to run. You can place as many commands as you need to produce the effect you want. You can run multiple filters, add layers, tweak layers, anything available in the procedure browser can be placed here.

As noted on the tutorial site this script uses the file-glob plug-in so we can pass in all our images. I’ve also changed the parameters for the image and drawable to the variables provided and added ‘blurnum’, so we can specify the number of pixels to blur on the command line in case we want to tweak it later.

Now we test it. The script needs to be moved into the scripts directory, I put mine in ~/.gimp-2.8/scripts. I like to keep them in a separate dir so I can version and edit it easier, so I actually place a symlink to the script in there.

Let’s do this!

Alright, we’re ready to go. Run this command and let’s see the magic happen!

gimp -i -b '(batch-graphic-novelfy "*.png" 6)' -b '(gimp-quit 0)'

Annnnnnnd, wait. What?

batch command experienced an execution error:
Error: ( : 1) eval: unbound variable: pattern

Oops. I axed the pattern parameter out of the first line of the script. It should be:

(define (batch-graphic-novelfy pattern blurnum)

Let’s try this again.

Animated Martians
Take me to your leader!
You can throw together a quick gif with something like this:
convert -delay 1x8 *.png -coalesce -layers OptimizeTransparency animated.gif

Now we can put this all together in a bash script.

# graphic_novelfy.sh
# Extract frames from video
ffmpeg -i $1 -r 6 -s 640x480 image%5d.png
# Run frames through gimp filters
gimp -i -b '(batch-graphic-novelfy "*.png" 6)' -b '(gimp-quit 0)'
# Create video with processed frames
ffmpeg -r 6 -i image%5d.png -vcodec qtrle -r 29.97 -pix_fmt rgb24 ${1}-novel.mov
# Clean up
rm image*.png
# Add audio in from original video
ffmpeg -i $1 -i ${1}-novel.mov -vcodec copy -acodec copy -map 1:0 -map 0:1 -shortest ${1}.final.mov

BAM! We’re done.

Run the script by passing in the video file you want to process and it spits out your processed video. If you’re not to familiar with bash scripting the first parameter is accessed in the script as $1 or ${1} if you need to separate it from an adjacent string.

The only new command in the script is the ffmpeg command to combine the audio from the original file with the new video. The -map parameter tells ffmpeg which streams we want to take from the two input files. To find the correct streams just pass all your input files into ffmpeg without any further arguments like so: ffmpeg -i video1 -i video2

Done

Simply running a gaussian blur and one filter is just barely touching the surface of what we could do to a video.

The downside is that the video will have to be processed before editing, and running each frame through gimp will take some time. If you’re running 24 or 30 fps video through gimp you’ll want to test your effects on small clips before taking the time to run it through anything large.

The upside is the sheer amount of possibilities this opens up for video effects. Anything you can automate in gimp to manipulate an image can be used on a video.

See the final clip below.

Icey-u Painting Cory Gugler

Ludum Dare 28 – Make a game in 48 hours

Ludum Dare is an event where people all around the world attempt to make a game in 48 hours, based on a theme voted on and announced at the start of the competition. There are no prizes, it’s more of a challenge to generate ideas and get people to make games.

I’ve wanted to participate in Ludum Dare for a couple years now and it just happened to fall on a completely free weekend for me this time, so I made my first HTML5 game.

Well, I consider it a prototype at this point. I used an HTML5 framework called Phaser, which I’ve had my eye on for awhile but haven’t worked with before. I spent a ton of time learning the framework through the excellent documentation and examples on the Phaser website.

I feel it was a success as now I have a working game prototype, instead of just a vague idea in my head. There is plenty of work to do but I feel it was definitely a worthwhile way to blow a weekend.

Check out my entry on the Ludum Dare website here:

http://www.ludumdare.com/compo/ludum-dare-28/?action=preview&uid=15070

 

P.S. – I’m running off about 10 hours of sleep over the weekend, please excuse the rambling nature of this post.

Flatfall - Watercolor painting

Refactoring Incremental Reader, or How to highlight text across nodes in the DOM

The last few days, I’ve been able to carve out some time to get back to working on Incremental Reader. My short term goal is to get a more sane structure so I can go back and add unit tests. I was recently exposed to Test Driven Development through a Coursera class “Introduction to Systematic Program Design“, and I’m pretty excited to figure out how to work this into my development process.

Highlighting text! As easy as 1,2…

The last feature I implemented was the right-click context menu to highlight selected text in the current browser tab. Correctly inserting tags to create the highlighted text was a huge pain, mainly because I want notes(highlighted text) to span multiple DOM elements and split anywhere (including the middle of an element). It’s been a bit since I’ve worked on the extension, so I thought I’d walk through the code to refresh my memory on how I implemented this feature.

First to get the selected text:

var highlightRange = window.getSelection().getRangeAt(0);

This returns a Range object, which can contain full nodes and parts of text nodes. The range object has a startContainer and an endContainer property which indicates the node at the start and end of the range. When these are equal, our range doesn’t cross through other nodes, making our job easy. All we have to do is split the node at the start and end offsets, and wrap the new node with with any tag containing a background(highlight) color.

var startContainer = highlightRange.startContainer;
var endContainer = highlightRange.endContainer;
var newNode = document.createElement('mark');
newNode.style.backgroundColor = "yellow";

if (startContainer == endContainer) {
    var splitNode = startContainer.splitText(highlightRange.startOffset);
    var temp = splitNode.splitText(highlightRange.endOffset);
    var insertElement = startContainer.parentNode.insertBefore(newNode, splitNode);
    insertElement.appendChild(splitNode);
} else {

First, we get the nodes where the range starts and ends. The .startOffset property of the range object contains the offset from the start of the node where the highlight began. .splitText() breaks a node at a given offset and returns the portion after the offset as a new node. The .endOffset is actually still remains correct (it’s updated on splitText), so we don’t have to worry about recalculating that value, we simply call it again to split the node at the end of the highlight.

Screenshot Highlighting a single DOM Node

splitNode now contains the node we want to visually highlight on the screen. To do this we create a <mark> element, style it, and insert it in the correct location. This is as easy as just getting the parentNode of our splitNode, and using insertBefore to insert it as a child element, just before the splitNode.

We then use .appendNode to move the node into the new <mark> element. If you use .appendNode on an existing element in the DOM, it will remove it from it’s current position and add it to the new location.

Now we’re done and that was pretty simple, right? Here’s where it gets a little more tricky. Imagine you want to start highlighting a note in the middle of an <h3> and end a few elements away on another <h3> or a <p> element. We can’t just split and wrap the entire block with one tag, it will break the DOM structure.

Screenshot - Highlight across multiple DOM nodes

The solution I came up with is to split the first element and the end element, then wrap all the elements with <mark> individually. Since the <mark> element is just for the user to have visual confirmation of the notes they’ve highlighted on the page, I have no problem with doing it this way. I also haven’t thought of any other solutions, so there’s that too.

After we have split the start and end node, we extract just the text nodes, since these are what we want to wrap. The only way to do this is to extract the the text nodes of range.commonAncestorContainer, meaning the closest element that wraps around the entire range.

Now we’ve all the text nodes including those of the parent container of our highlighted text. To get rid of all the extra nodes, we can use Range.intersectsNode to determine which nodes in our list are actually in the selection. Then, we just loop through our nodes and wrap them all with a <mark> element. You can check out the full code here: content.js on Github

Screenshot Successful DOM highlighting

Refactoring

I’ve got way too much going on in the popup code. I’m going to try to consolidate everything I can into the event script. I’m not sure if the popup page is even going to stay, as the plan is to have a management view to manage articles and notes. The popup might be redundant, as saving pages can be done with a context menu instead.

I’ve also been working out a new way to structure the data. Right now all pages are stored as an array in localStorage. Each item in localStorage is an array containing the url, page title, and the current vertical location of the window (scrollTop or pageYOffset ). This was fine for testing but I plan to store much more info.

Here is the current working list of how I plan to restructure it:

// Collection of articles, future possibility to group articles
articleCollection {
    articles:        // list of articles

    // methods
    save:            // save to local storage
    load:            // retrieve from local storage
}

article {
    url:             // url of article
    prevLocation:    // scrollTop or pageYOffset
    notes:           // list of notes
    created:         // date article added
    accessed:        // date last accessed
    archived:        // bool, allow user to set when done reading

    // methods
    save:            // save article to collection
    remove:          // remove article from collection
}

note {
    content:         // raw text content of note
    location:        // original DOM node position, may help
                     // when locating note on page reload

    // methods
    save:            // save note to article
    remove:          // remove note from article
}

You can see I’ve tried to structure everything in a little bit more of an object oriented way. I’ve added some possibilities for helper methods, but haven’t added getters. I’m not sure that I need to hide any of the data (i.e. wrap in a closure), but I’ll consider that in the future.

You can see I’ve got a pretty good start on getting this up and running, but I’ve still got some more planning to do. I haven’t even gotten in to adding unit tests yet!