Keith Bentrup’s Portfolio

I’m on a simplification kick, and I’ve been dramatically reducing the clutter in my life and enjoying the results.

I recently noticed a good deal on a drive at Microcenter and thought “Why keep 5 external drives (1.3 TB total, 5 usb cables, 5 power cords) when a single drive (1.5 TB, 1 usb cable, 1 power cord) will do?”

So began the long process of consolidating, organizing, and erasing years worth of files, and if you’re doing the same, I have a few tips to share.

After consolidating everything to one disk, I wanted to find duplicate files that may have proliferated over the years. Fortunately, there’s a little utility called fdupes that does the trick nicely. On a Windows system, you can use it through cygwin.

fdupes -r /cygdrive/VOLUME_HERE

The first pass was preliminary to see how bad it was. I found 100s of GBs worth of data that I had forgotten about. There were also several .svn directories cluttering up the results. To remove that useless (for a backup) metadata, I removed it with

IFS=$(echo -en "\n\b")
find -regex ".svn" -delete

The first line sets the input file separator to a newline character (as opposed to the space) so the results of the find in the next statement works as expected. Now it will treat filenames with spaces as a proper path.

Now I could recursively, forcibly delete all the duplicates:

fdupes -rFd /cygdrive/VOLUME_HERE

I also wanted to remove all those Thumbs.db files that Windows loves to create as well as all the empty files, (now) empty directories, and old files

find -iregex ".*(Thumbs\.db|\.tmp|\.old|\.bak|\.dmp)$" -delete
find -size 0 -type f -delete
find -type d -empty -delete

Note: You may want to run these commands without the delete option first as a sanity check.

Now prep the old drives for donation …

Considering that you may have very sensitive data (e.g. passwords, bank/credit card statements, etc.) on your old drives, be sure to clean the drives before you donate them.

I simply formatted each and then used the windows built in cipher utility to clean each drive after copying over the contents.

cipher /w:VOLUME_HERE:

This wipes all the empty space on a disk with a 3 pass system – writing all 0’s, writing all 1’s, and then writing random data. In all that’s 4 passes (including the original format).

I could go on to discuss encryption of your data on your new drive and offsite backup, but that’ll have to wait for another day …

The fastest download is no download, so I was trying to prevent even the 304s (last modified requests & responses) on a recent project. It had been a while since I had to configure Apache for this, but here was the start of a .htaccess file:


<FilesMatch "\.(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf)$">
ExpiresActive On
ExpiresDefault "access plus 10 years"
</FilesMatch>

# compress text, html, javascript, css, xml:
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript
Header unset ETag
FileETag None
Header unset Last-Modified


I set up a versioning system on the backend so that whenever resources are updated new version numbers are automatically generated to create new request URLs. Thus, I could set far future expires tags. However, it didn’t seem to be working. I kept seeing the requests in fiddler even though the Expires header was set and the Last-Modified header removed. The problem? I was hitting refresh which was causing the clients (FF & Chrome) to request the resources regardless of the headers. Navigating away and then back generated the expected behavior. :p

wget can be a very handy tool during development. Combine wget with other *nix utilities (like time and cron), and you can write some very useful scripts/snippets. For example,

wget -p -H -nv http://www.my-domain.com/cart/ 2>&1

uses -p/–page-requisites to download all the files necessary to render the page while -H causes it to span domains if necessary. If you pass this to grep (after redirecting the output (defaulted to STDERR) to STDOUT), you could look for 403/404’s or any other undesirable response code. The -nv/–no-verbose simply prevents the deluge of information that wget spits out by default.

Here’s part of a shell script run through my crontab that I have set up to time downloads during the development cycle. I can monitor a list of urls on my dev/test servers (passed in $1) and monitor the performance over time.

for i in $(
 # list of urls here
 echo '/product/index.jsp?productId=3633213'
 echo '/cart/index.jsp'
 # ....
); do
time -f "`date` | %E | $1$i\n" -a -o time.log wget -a wget.log -O /dev/null -p $1$i
done

So I was having problems backporting some changes to a fork of our platform today. WinMerge kept reporting files as different in the summary view, but displayed no differences in the file comparison view. I looked closer and noticed the character sets were “different”.

A developer on the team had started saving files in the repository with BOM (Byte Order Mark) information. I don’t know if it was intentional or an effect of their IDE. Either way, it had to go, so … perl to the rescue.

for i in $(find -regex ".*\.jsp$"); do perl -pi.old -e 'BEGIN { $/ = undef; } s/^\xEF\xBB\xBF//' $i; done

I sometimes keep and share development related scripts in the repository, but by default, svn will not remember file modes. To set the script to execute, simply type:

svn propset svn:executable ON myfile.sh

In TortoiseSVN, just set a new property on the file: