I’ve been studying grammars and formal languages, among other things, this past semester in my Theory of Computation class. One thing that it reminded me of was the second Javascript application I ever developed (with the help of my friend and college classmate Miranda Tarrow): The Random Goth Lyric Generator.

I took a simple sentence structure (subject, verb, adjective, object) and made random substitutions for each line, 4 lines per stanza, 4 stanzas per poem.

The (slightly less-than-formal) grammar for each line would look something like this:
Line -> SSL | PSL
SSL -> SNP SV A O
PSL -> PNP PV A O
SNP -> Singular Noun | Singular Noun Phrase
PNP -> Plural Noun | Plural Noun Phrase
SV -> Singular Verb
PV -> Plural Verb
A -> Adjective
O -> Object

We could extend this to the entire poem:
Poem -> Stanza Stanza Stanza Stanza
Stanza - > Line Line Line Line

The word list was meant to be dark and foreboding but was often hilarious–the examples included:

Nouns & Noun Phrases:

• My solitude
• Your touch
• A ravenous she-wolf
• Spiders

Verbs & Verb Phrases:

• entangles
• summons
• grovels before
• spews forth

Adjectives:

• labyrinthine
• diseased
• spectral
• infernal

Line -> SSL -> SNP SV A O could become:

Your touch entangles infernal spiders.

I don’t know why the list of objects was a separate list of nouns, as it seems to me now that it could have pulled from the same list. Since the grammar used just one sentence structure, the results were very repetitive but frequently humorous. I often considered expanding the possible sentence structure (something as simple as making the adjective optional), but decided that the repetition was part of the charm. In fact, many poems and song lyrics feature repetition, and the results seemed eerily intentional at times.

The page was very popular for a time. I received quite a bit of e-mail regarding the page, including suggestions for additional words. Someone sent a song they’d recorded for which they used the random lyrics (with the addition of a shouted, “There’s that m———— word again!” in the middle). At least one randomly-generated poem was published in a small poetry journal.

I’ve considered creating a sequel to parody William Carlos Williams and loading it up with words from his own poems:
Poem -> S1 S2 S3 S4
S1 -> Noun Verb newline Preposition
S2 -> Article Adjective Noun newline Noun
S3 -> Adjective Preposition Adjective newline Noun
S4 -> Preposition Article Adjective newline Noun

Which might produce something like:

no one spilled
with

the whole honey
suckle

pressed after sweet
odor

while the urgent
petals

It just doesn’t seem quite as funny or compelling. I can venture a guess that William’s sparse form and carefully selected language doesn’t lend itself to random imitation as well as verbose and self-indulgent free verse. Although perhaps the sample is merely too small!

To my knowledge, the best tool for the job is FFmpeg, an open source video tool. FFmpeg provides much of the underlying processing functionality for other popular video tools, such as Handbrake, Miro, and MPlayer. Since you compile FFmpeg from source code, it should run on any system with a C compiler, including my OS X box. Unfortunately, it’s not the most user-friendly software package in the world.

The developers recommend using the latest version from Git (which finally forced me to install Git, something I’d been meaning to do anyway). One how-to for FFmpeg on OS X docs suggest that I’d also need LAME in order to process audio. (The audio is irrelevant for my use case, so I didn’t bother with LAME.) But I couldn’t compile FFmpeg because, apparently, OS X doesn’t include GCC. To get GCC on OS X, the official Apple way, I needed XCODE from the Apple Developer Tools. To get those you have to sign up for an Apple Developer account. Welcome to Dependency Hell, now featuring bureaucracy!

Hours later, I’ve compiled FFmpeg. However, the videos I’m dealing with are raw H.264 video files from a Night Owl K-44500-C surveillance system. Also, I want to save them as H.264-encoded MP4 files. That means I needed additional H.264 support–or, at least, I thought I did–from the x264 project. And you need YASM to compile x264.

I was actually unable to compile YASM from the Git repository, although I was able to compile it following the instructions in this Handbrake for Mac OS X guide.

I recompiled FFmpeg with the –enable-libx264 and –enable-gpl switches:
./configure --enable-libx264 --enable-gpl
make
sudo make install

To select just the relevant portions of the video, I used the -ss (start/seek position) and -t (time/duration) flags, e.g.:
ffmpeg -f h264 -i input-video-file.264 -ss 180 -t 8 output-video-file.mp4

The above example takes 8 seconds of the input video starting at the 3-minute (180-second) mark.

However, when I played back the output, it played much too fast. The source videos included a timestamp, and about 4 seconds ticked by every for every second of video! It turned out that the source videos were recorded at 7 fps (frames per second). I added a flag to specify the framerate:
ffmpeg -f h264 -r:v 7 -i input-video-file.264 -ss 180 -t 8 output-video-file.mp4

After running this a few times for different 8 second segments, I needed to put the segments back together again. This is such a relatively common use-case that FFmpeg has instructions in their FAQ, How can I join video files?. The first method–concatenating MPEG-2 files–seemed like the easiest option. However, MPEG-2 doesn’t support the framerate (7 fps).

I tried the other suggested method of concatenating videos, using named pipes. This worked, although the BASH script was convoluted and very particular.

Another thing I wanted to add to the video was a separator–some static frames to divide each 8-second clip–and a title card. At first I created a JPEG and turned it into a 1-frame video, concatenated it with itself to create a 2-frame video, and then 4, 8, 16, etc. However, I discovered a much easier method of creating a video from a single image using the loop flag:
ffmpeg -f image2 -loop 1 -r:v 7 -i image.jpeg -pix_fmt yuv420p -an -t 2 image-movie.mpeg

(The pix_fmt flag was to set the correct color space, and the an flag ignores the audio channel.)

Now I had title cards, clip separators, and 8-second video clips that I could combine into a single video. But I needed to do this hundreds of times! I wrote a Python script to generate the appropriate BASH script based on the input filename. The BASH script would create the title cards and clip separators using ImageMagick, and then call the appropriate FFmpeg commands to create and concatenate the video.

The ImageMagick commands look like this:
convert -size 704x480 -background SteelBlue1 -fill black -font Helvetica -pointsize 72 -gravity center label:[Video Title] titlecard.jpg

Then I used the find command to run the Python script on all the video files:
find *.264 -maxdepth 1 -exec ./process.sh '{}' \; -print

Eureka! It worked. Almost perfectly.

Almost.

Some (but not all) of the output videos would produce solid gray frames after a certain time point. Reviewing the FFmpeg output for those files, there was an error:
[h264 @ 0x9460340] FMO not supported

FMO stands for Flexible Macroblock Ordering, and based on the response to a libavcodec issue from 2009, the FFmpeg developers don’t plan to support it (although one of the developers suggested that, if someone would like to create a software patch to enable FMO support, the community would welcome it).

I wrote to technical support for the camera system and asked if they had any suggestions. They replied that, although conversion to MPEG formats was not supported, they do provide a conversion utility to convert to AVI. I was able to successfully convert the AVIs to MP4s. An annoying extra step, but one that is only necessary in a subset of cases.

This solution took me several weeks to figure out, although it should save quite a bit of time in the long run.

The basic thesis of the lecture was that good algorithms matter and that we need to champion good algorithms where they are most needed (particularly in the sciences).

One of his points was that computer science is currently very abstract and lacks a basis in the scientific method. Algorithms need to be tested against models to see how they actually perform. In some cases, the theoretical performance of an algorithm can be off by several orders of magnitude compared to actual performance. For example, the quicksort algorithm is quadratic (N²) in the worst case, but N log N most of the time. There’s a reason why quicksort (by the way, the subject of Sedgewick’s 1975 PhD dissertation) is widely used, in spite of the fact that it is O(N²) versus binary sort’s O(N log N).

Sedgewick said, though, that he has run into many computer scientists who fail to observe the difference between theoretical worst-case and actual performance. Some will choose an algorithm based on Big-O analysis alone. Sedgewick’s response: Big-O is an upper bound, but is your input an example of the worst-case? Probably not. Algorithms should be chosen based on their actual performance.

[As an alternative to Big-O notation, Sedgewick suggested Tilde notation, although from my perspective I don't see that there is a great difference between them.]

He also gave an example of taking theory too far in the other direction. A computer scientist gave a talk demonstrating that his algorithm, Algorithm B, though exceedingly complex, was superior to the simpler Algorithm A. When Sedgewick asked him why, he explained that Algorithm B removed a log log N factor. Sedgewick’s analysis was that log log N, in this universe, amounts to 6 — hardly worth trading algorithms for what, realistically, amounts to a constant factor.

[Why 6? Wikipedia and other sources estimate the number of atoms in the observable universe at 10⁸⁰. The natural logarithm of 10⁸⁰ is 184. The natural logarithm of 184 is 5.2. 6 sounds like a fine estimate.]

Another point was that scientists often need algorithms in their daily work, but do things the hard way for a lack of knowledge. One example was a biologist who was trying to use Excel to calculate a standard deviation for over a million data points, an idea that caused several audience members to cringe.

How do we bring a better understanding of algorithms to the masses? (By masses, I think he really means the masses of college-educated scientists–not quite everyone, but still a much larger group than just computer scientists.) He had several suggestions:

Analytic Combinatorics
From what I gather, analytic combinatorics is a way of using formal languages to describe recurrence relations, and thus a simpler (and easier-to-teach) method of creating generating functions. I don’t exactly know what that means, but you can read the book on it (by Flajolet and, of course, Sedgewick): Analytic Combinatorics (PDF).

Testing Algorithms Empirically
In computer science classes, he suggests students run a program on an increasing series of inputs (e.g. n = { 1000, 2000, 4000, 8000, 16000, … }) and examining the ratios of input size to run times to understand the real impact of running in linear time, N log N time, quadratic time, etc. (This is something that some of my past computer science courses have included, so apparently they have already adopted this piece of Sedgewick’s advice.)

Changing Intro to CS Courses
Sedgewick recommends identifying core elements of computer science, such as classic algorithms, and teaching them to everyone as early as possible. Some changes made to the curriculum at Princeton (where Sedgewick teaches) have led to a dramatic increase in enrollment in intro computer science courses and from a wider range of majors.

Change Publishing
Sedgewick touted his Introduction to Programming in Java and its accompanying web site as a major change for textbooks. The programming examples are short and simple, but demonstrate a wide range of real-world applications across several branches of science. (Although he did not mention it in the lecture, I also appreciated that the examples in this book often include graphics, sound, and animation — which are far more thrilling results than the usual ASCII that intro CS students see as the fruits of their labors.)

He also criticized academic publishing for making journal articles look as much like boring print articles as possible, in spite of the fact that they are now primarily accessed online. Where are the full-color figures? The hyperlinks? Animated simulations? These things are all possible online, but instead publishers restrict content to the form that is least likely to be accessed.

Several times in the lecture, he also mentioned that freshly-minted computer scientists often have little or no background in science: no physics, no chemistry, no biology. Although he recognized this as a problem, he had no solutions (other than to, perhaps, require foundational science courses as part of a CS degree).

The last two items–changing the curriculum and publishing–really sounded like a Sedgewick paid-programming infomercial. “Everyone should take courses in the subject of which I am an expert, and they should use the book I wrote to teach it.” It was hard not to be a little skeptical of his motives. At the same time, I can’t help but think that he’s right.

Set-UserInterface2010.ps1 -url "https://my.sharepoint.site/path/"

Today I received a request to update a list of 32 sites. After updating one, I thought–this is going to be tedious. We can improve this.

First, I updated the parameter to accept an array of strings instead of a single string.

Before
param([string]$url = $(throw "Please specify a Site URL to convert to the SP2010 look and feel"))

After
param([string[]]$url = $(throw "Please specify a Site URL to convert to the SP2010 look and feel"))

Next, I wrapped the call to the main function in a ForEach loop:
ForEach ( $siteurl in $url ) {
...
# Throw in a line break to separate output
Write-Host ("`n`n")
}

The script can still take a single site as input (the string is treated as a string array with a single element), but now I can also pass it a list:

Set-UserInterface2010.ps1 -url "https://my.sharepoint.site/path1/","https://my.sharepoint.site/path2/","https://my.sharepoint.site/path3/"

Here’s hoping that taking a couple minutes to update the script and running it once saved me more time than running it 32 times!

Thinking, Fast and Slow

For example, programmers have been arguing since the 1970s about the placement of curly braces. Some people prefer:

if ( true ) {
doSomething();
}

Others prefer:

if ( true )
{
doSomething();
}

Crockford says that if the compiler treats these 2 forms as equivalent, then there is really no difference (so long as you are consistent). These are Gut decisions. However, people will use their Head to try to rationalize their Gut decisions and come up with some ridiculous rationalizations.

OK, fine. What does that mean in practical terms, i.e. writing code?

Crockford's advice is:

Prefer forms that are error-resistant.

This is aligned closely with my own advice:

Prefer explicit forms to implicit forms.

One of Crockford's illustrations (which I have not verified but sounds plausible enough) is that the space character was introduced into Latin writing after the prior practice of running all the words together led to ambiguity and transcription errors. The space characters made the text more explicit, and therefore less prone to transcription error.

Here are 3 of the Javascript examples Crockford provided during his talk:

Curly Braces & Conditional Statements and Loops
In Javascript, a conditional statement or a loop containing only one statement can be written without curly braces:
if ( condition ) doSomething();

This form is error-prone. Include the curly braces:
if ( condition ) {
doSomething();
}

Some people might complain and say that this is less concise and requires extra typing. Crockford's response was: less concise is better, and the majority of the time it takes to write a program is in thinking, not typing. If adding a few characters is slowing you down, you either need to take a typing class or spend more time thinking!

Double-Equals versus Triple-Equals
Crockford said that using double-equals (==) to compare objects is always a bad idea because it does implicit type conversion. For example:
var n = 5;
n == 5; //true
n == "5"; // true
n === "5"; // false
n === 5; // true

The triple-equals also helps developers know that a number is expected, not a character or other data type.

As Crockford emphasized:

If a feature is sometimes problematic, and a more reliable feature exists, use it!

Some people might try to justify the use of implicit type conversion: "But I'm doing it intentionally!" Crockford said that relying on implicit type conversion means you don't know what you're doing.

Incrementing Values
Crockford said he never uses the prefix or postfix ++ operator, because many people don't understand the difference and it can lead to confusion. For example:
var x = 5;
var y = x++; // y is now 5 and x is 6
var z = ++x; // z is now 7 and x is 7
x += 1; // x is now 8. This is Crockford's preferred method to increment a variable

Crockford provides several resources to help people write better, more error-resistant Javascript:

• Code Conventions for the JavaScript programming language
• JSLint, which alerts you to errors in your code or error-prone code ("JSLint will hurt your feelings")
• JSMin, which "minifies" (removes unnecessary characters) from Javascript files. This should be largely unnecessary if your web server uses Gzip compression.

I’ve been interested in unit testing and test-driven development for several years now, but have never put it into practice. It seemed like a good idea in theory, but it required picking a testing framework, installing it, configuring it, and, of course, writing good tests. Without any hands-on experience, it’s a difficult practice to adopt. Getting introduced to unit tests this way lifts nearly all those burdens. On top of that, the advantages are clear: the tests will tell you when you have it right.

I was a little surprised to run into unit testing via coursework, because I’d been under the impression that computer science education focuses a lot on theory, and skips over a lot of the practice. I’ve met a lot of people coming out of computer science programs who, while probably excellent theorists, don’t know heads from tails when you sit them at a terminal. I was happy to see that my coursework included practical knowledge as well.

All the unit tests are provided by the professor. I assume we may write some tests of our own later on–and certainly nothing is stopping us from writing our own tests now–but I do wonder how many students will be prepared to take that next step. From what I understand, writing good tests is the better part of successful unit testing.

Some things I’ve noticed about the provided unit tests:

• There are a lot of them. The unit tests are half as long the code that passes them.
• There are many sample values to test the same function, mostly to test specific edge cases. Failure at a specific edge case helps to identify where the code went wrong.
• In some cases, the tests are within a loop that generates random test data. Although there’s nothing quite like real human input to break your programs, 1000 random tests might help.

In practice, do programmers tend to write their own unit tests? I imagine it would be ideal if you partnered with someone, and you wrote their unit tests, and they wrote yours. It might be easy to overlook an edge case or dismiss something as impossible if you are too involved with the specifications. At the same time, it is probably difficult to write a unit test without spending some time reading the specifications and understanding exactly what it is supposed to do.

I had always heard that, unless the image conveys important information (e.g. a graphic of text used as navigation, or a chart or graph) that the alt text should be left blank:
<img src="myimage.png" alt="" />

A screen-reader passes such an image over without saying anything. This makes sense to me. When I’ve closed my eyes and tried navigating the web using a screen-reader like JAWS, anything non-essential was a distraction and just got in my way. Knowing that a page contained an image of, say, a corporate headquarters in no way helped me understand the page content.

However, a colleague of mine suggested that the alt text should describe, briefly, the image. He offered a compelling use case: you are browsing on your mobile device with images disabled, due to bandwidth. How would you know if there was an image that you did want to view if no description was provided? This is a case where the alt text can provide a better user experience for users without vision impairments. But does it make the experience worse for users with vision impairments using assistive technology?

(A case could be made that vision-impaired users should also know there is an image on the page for orientation purposes: “The link to the annual report is underneath the photo of the corporate headquarters.” However, what may appear above or below on screen may not make sense when page content is read linearly.)

I looked to the WCAG 2.0 section on text alternatives, which states that images used for decoration or formatting should be implemented in such a way that they “can be ignored by assistive technology.” That’s a good case for using an empty alt attribute. If the image is sensory (WCAG 2.0 has been criticized for being vague–obviously anything visual is sensory), then the item should “at least provide descriptive identification of the non-text content.”

What about that photo of the corporate headquarters then? It’s decorative, but not in the same way as a fleuron or a border. It may not be an inspiring image, but maybe it should have associated alt text.

I decided to check 4 sites that I thought might demonstrate best-practice, but found little consistency across these examples:

• The National Federation for the Blind – they use rather extensive alt text for the main image on their homepage: “Graphic consisting of two photos. On left is a group of children with white canes on a hayride. Right is a close-up of a finger reading Braille.” However, they fail to use the alt attribute for their menu divider graphic, which is clearly a decorative element.
• Freedom Scientific’s JAWS Screen Reading Software – the alt text “A student uses JAWS to do work on a desktop computer” accompanies a photo of a man at a computer. Decorative images (menu dividers, stars) use an empty string for the alt text.
• WAIM – Web Accessibility in Mind – they avoid the issue on their services page by inserting the pictures as CSS background images. These would not appear at all to a screen reader, to a text-based browser, or to a user agent with images disabled. This would be functionally equivalent to using empty alt text.
• The Social Security Administration’s Disabilities Benefits – this page gets it completely wrong, including alt text for images that do not even appear visible to users with normal vision (e.g. a tracking image with the alt text “DCSIMG” and a spacer image with the alt text “blank space”).

MIT’s general web-accessibility guidelines offer some additional guidance:

ALT tags are often misused, mostly people overuse them. It’s better to leave the ALT tag blank (ALT=”") then to enter a text description that’s not useful or is redundant. For example an image with a caption below it does not need alt text that matches the caption, leave the alt text blank to avoid redundancy.

The University of Michigan’s Accessibility Quick Guide suggests using empty alt text for non-informative images.

Unfortunately, we’re still left with a rather vague recommendation: use a description when useful or informative. How do we decide when a description is useful or informative? My gut feeling is to agree with MIT: it should be left blank in most cases (such as with the hypothetical photograph of a corporate headquarters), but I think no great harm is done if a brief description is included.

The best and easiest way I’ve found to do this is via Cygwin:
touch -t 201109201145 *.pdf

When I searched for how to accomplish this, I found all sorts of sites advising me to download a $20 shareware application, or write a C# application to do it. Fortunately, in a *nix type environment, it’s a one-line command.

There are other ports of Unix/Linux/*nix commands for Windows, some of which may be more lightweight than Cygwin. (I rely on Cygwin quite a bit, so I have it installed on all my Windows machines.) I have not used any of the following, but here are links to some alternatives, in case you are interested:

• Core Utils
• GNU Win32 Core Utils
• Unx Utils
• Touch for Windows

Sys.WebForms.PageRequestManagerServerException: An unexpected error has occurred

I found a couple blog posts about this suggesting a permissions issue (which made sense, as users in other groups could successfully edit pages):

• SP2010/Wiki Pages: “Sys.WebForms.PageRequestManagerServerErrorException: An unexpected error occured.” when editing a page
• Sys.WebForms.PageRequestManagerServerException error when editing pages

Both sites suggested that an Access Denied message should appear in the ULS logs. However, no such message appeared in relation to my errors.

ULS entry:
System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.SharePoint.Publishing.WebControls.MediaWebPart.get_WebPartAdderId()
at Microsoft.SharePoint.Publishing.WebControls.InsertMediaRibbonButton.RegisterRequiredScripts()
at Microsoft.SharePoint.Publishing.WebControls.InsertMediaRibbonButton.OnPreRender(EventArgs e)
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint... 1317dffd-b607-4f1c-9145-b4adc1d364ba

The MediaWebPart. SharePoint 2010 added a video/audio insertion feature on the ribbon, and the error occurred when trying to add the button.

It turns out, the permissions to the Web Part Gallery had been customized (i.e. broken inheritance from the parent site collection), and some groups who had permission to edit wiki pages did not have read permission for the Web Part Gallery (and, consequently, the MediaWebPart). Adding Restricted Read access for the group in question (to either the MediaWebPart or to the Web Part Gallery) solved this problem.

I don’t imagine this will be a common problem for other people, but I thought it worth mentioning.

• 2,001
• 1,998
• 2,011

This is confusing to the viewer, as values no longer look like years.

The most frequent suggestion is to create an additional field that calculates a value based on the Number field and ignores all non-numeric parts. This is relatively straightforward, and can be accomplished entirely within the browser:

=TEXT([MyNumericData], "0")

Or, in a case where you’d like to retain two digits after the decimal:
=TEXT([MyNumericData], "0.00")

SharePoint appears to store the data in its numeric format in a another column, hidden to the browser but visible in SharePoint Designer. It’s [column name] followed by a period. For example, I created a field called MyNumericData, so the hidden field is MyNumericData. Again:

• MyNumericData
• MyNumericData.

Adding this to a view via SharePoint Designer is non-obvious. Here are the steps I used:

1. Click on a data item in Design mode until you see a tab labeled xsl:value-of:
   
   

   SharePoint Designer 2010 Design Mode

2. Right-click the item and select Edit Formula
3. Remove the current XPath expression
4. Add the MyNumericData. row from the fields pane
5. Add the format-number function from the Math / Number functions
6. Add the appropriate format pattern (e.g. ‘#’)

The appropriate format pattern depends on how you wish to display the data:

• Display integer value:
  format-number($thisNode/@MyNumericData.,'#')
• Always display two digits after the decimal:
  format-number($thisNode/@MyNumericData.,'0.00')
• Display up to 2 digits after the decimal:
  format-number($thisNode/@MyNumericData.,'#.##')

If you look at the code view after making these changes, you will see that SharePoint Designer added a hefty chunk of XSL to the XsltListViewWebPart.

Is there any advantage to this method over adding a calculated field? Not as far as I can tell–but it’s interesting to note that there is more than one overly-complicated solution to what seems like a trivial problem.