VMWare VSphere CLI vmware-cmd and the cfg parameter

I have a VMWare ESXi host. I can manage it through VMWare Fusion, although the options seem limited (I’m used to using vCenter Server, but I don’t have the license for that in this environment). I thought I’d give the VMWare vSphere Command Line Interface (CLI) a try. This was a mistake, but if you insist on following me down the same path, see the Drivers and Tools section on the VMWare vSphere Downloads page to get started.

First I tried vmware-cmd.

C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd
'vmware-cmd' is not recognized as an internal or external command,
operable program or batch file.

The actual file is vmware-cmd.pl (it’s in the bin folder).

I was able to run one command, to list the virtual machines on the host:
C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl -H esxi.osric.net -l
Enter username: chris
Enter password:

The documentation I was looking at was probably outdated, as the newer documentation gives better examples. But the version I was looking at indicated that most of the other commands require a <config_file_path> or <cfg> parameter. Unfortunately, it does not specify what those values consist of or what they might look like. There was a hint in the docs in vmware-cmd Overview:

vmware-cmd is a legacy tool and supports the usage of VMFS paths for virtual machine configuration files. As a rule, use datastore paths to access virtual machine configuration files.

It appears that <cfg> is the path to the VMX. There are several different ways to specify this:

Full path using GUID
C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl -H esxi.osric.net /vmfs/volumes/272c880d-a89548c1-a530-4bccbbad9507/benvolio/benvolio.vmx uptime
Enter username: chris
Enter password:
getuptime() = 7193

(The GUID is displayed in the output of the list of virtual machines from vmware-cmd.pl -l.)

Full path using Datastore Name
C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl -H esxi.osric.net "/vmfs/volumes/test vms/benvolio/benvolio.vmx" uptime
Enter username: chris
Enter password:
getuptime() = 7578

Datastore Name + relative path
C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl -H esxi.osric.net "[test vms] benvolio/benvolio.vmx" uptime
Enter username: chris
Enter password:
getuptime() = 7822

Entering my username and password every time is tedious though. According to the Connection Options for vmware-cmd:

The vmware-cmd vCLI command supports only a specific set of connection options. Other vCLI connection options are not supported, for example, you cannot use variables because the corresponding option is not supported.

In this case, I have the vSphere CLI installed on a password-protected Windows 2012r2 virtual machine, so I didn’t feel it was too much of a risk to set a temporary environment variable to store some of the connection options:

C:\Program Files (x86)\VMware\VMware vSphere CLI>SET VMOPTIONS=-H esxi.osric.net -U chris -P t0u6hpa55w0rd
C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl %VMOPTIONS% -l

Remember how the documentation said that “vmware-cmd is a legacy tool”?

I’m not sure what the official replacement is–possibly the PowerShell-based VMWare vSphere PowerCLI–but it turns out that the vSphere Client is free. Accessing your ESXi host via HTTPS should provide a link to download the installer. The vSphere Client does not appear to be something you can script against or automate, but for simple tasks it is much easier to use than vmware-cmd.pl.

Mail Users in Office 365 don’t have SMTP access

On-premises mail users (at least in Exchange 2010) had access to send mail as their organizational address through the on-premises SMTP server. However, mail users in Exchange Online cannot send mail as their organizational address using smtp.office365.com.

So what can we do?

In order to use SMTP, users need full mailboxes. But these users should not actually have mailbox access. As a test, created a mailbox and I disabled all email apps for the mailbox in Exchange Online:

An Office 365 user's email app settings
An Office 365 user’s email app settings

The test user was no longer able to log in to Outlook on the Web (also known as OWA). SMTP still worked. Email forwarding still worked (although the user would not be able to set the forwarding address themselves).

Creating a user mailbox requires a user license, whereas mail users do not require a license. If you have a lot of on-premises mail users that now need full mailboxes, this could be problematic.

Summary:
In Exchange Online, a UserMailbox with all email apps disabled is equivalent to an on-premises Exchange MailUser, except that the former requires a license.

Middleman error – `block in replace_gem’: middleman-cli is not part of the bundle. Add it to Gemfile.

I ran into an error while setting up a Middleman site on my PC. I already had Ruby and RubyGems installed, so I followed the instructions to install Middleman and start a new Middleman site:

gem install middleman
middleman init project
cd project

The next step was to start the preview web server, which produced an error:

$ bundle exec middleman server
DL is deprecated, please use Fiddle
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/bundler-1.7.7/lib/bundler/r
ubygems_integration.rb:256:in `block in replace_gem': middleman-cli is not part
of the bundle. Add it to Gemfile. (Gem::LoadError)
from c:/RailsInstaller/Ruby2.1.0/bin/middleman:22:in `<main>'</main>

Continue reading Middleman error – `block in replace_gem’: middleman-cli is not part of the bundle. Add it to Gemfile.

Downgrading a Debian package

After I updated a Debian Wheezy server to Debian Jessie, I was having some problems with VLC. I was using the cvlc to capture and record video streams, but the capture would fail after 2 seconds. A colleague suggested that I leave the OS version alone, but downgrade VLC to confirm that the new version of VLC was causing the problem. A sound idea, but how do you downgrade a Debian package?

I found a variety of helpful sites and came up with the following:

  1. Add the source for the downgraded package the apt config
  2. Specify the target release in the apt config
  3. Use “pinning” to tell apt to use the older versions for the package and its dependencies
  4. Use apt-get install to install the dependencies and the package

Details as follows:

Add the source for the downgraded package the apt config
In this case, I needed to add the
I left my /etc/apt/sources.list as-is and added the following to /etc/apt/sources.list.d/vlc.list:

deb http://mirror.cc.columbia.edu/debian/ wheezy main non-free contrib
deb-src http://mirror.cc.columbia.edu/debian/ wheezy main non-free contrib

deb http://security.debian.org/ wheezy/updates main contrib non-free
deb-src http://security.debian.org/ wheezy/updates main contrib non-free

deb http://mirror.cc.columbia.edu/debian/ wheezy-updates main contrib non-free
deb-src http://mirror.cc.columbia.edu/debian/ wheezy-updates main contrib non-free

I used Columbia’s mirror because it’s fast and geographically nearer than most of the other mirrors. I’m sure not all of those sources were necessary (there are no security-related packages in VLC) but it didn’t hurt anything to include them.

Specify the target release in the apt config
I added the following to /etc/apt/apt.conf.d/80targetrelease to specify that I want Jessie to be the target/default release:

APT::Default-Release "jessie";

Use “pinning” to tell apt to use the older versions for package and its dependencies
I created the file /etc/apt/preferences.d/vlc containing the following. I added to the list of packages as apt-get install failed due to missing dependencies. The pin-priority 1001 was suggested by several sites to be used only in the case of downgrading a package.

Package: vlc
Pin: release n=wheezy
Pin-Priority: 1001

Package: vlc-data
Pin: release n=wheezy
Pin-Priority: 1001

Package: vlc-nox
Pin: release n=wheezy
Pin-Priority: 1001

Package: libvlc5
Pin: release n=wheezy
Pin-Priority: 1001

Package: libvlccore5
Pin: release n=wheezy
Pin-Priority: 1001

Package: vlc-plugin-notify
Pin: release n=wheezy
Pin-Priority: 1001

Package: vlc-plugin-pulse
Pin: release n=wheezy
Pin-Priority: 1001

Package: xdg-utils
Pin: release n=wheezy
Pin-Priority: 1001

Use apt-get install to install the dependencies and the package
I started by trying to apt-get install vlc but it complained about missing dependencies. I added the dependencies to the preferences file as describe in the section above, and then was able to install the following packages from Debian Wheezy:

apt-get install libvlccore5
apt-get install vlc-data
apt-get install libvlc5
apt-get install vlc-nox

Alternatively, I believe I could have skipped the preferences file and setting the pin-priorities by specifying the target Debian version when running apt-get install:

apt-get install -t wheezy libvlccore5
apt-get install -t wheezy vlc-data
apt-get install -t wheezy libvlc5
apt-get install -t wheezy vlc-nox

However, I’m not sure that the desired package version would be preserved after running apt-get dist-upgrade.

References
The following sites were helpful to me while I was figuring out how to do this, and if you are interested in pin-priority and the different values to use in different scenarios, I definitely recommend the first link:

Applying per directory X-Frame-Options headers in Apache

To help prevent against click-jacking, I had applied the following to my Apache 2.2 configuration based on the suggestions described in OWASP’s Clickjacking Defense Cheat Sheet and Mozilla Developer Network’s The X-Frame-Options response header:

Header always append X-Frame-Options SAMEORIGIN

However, my site has certain pages that are included in an iframe on another site, for the purpose of displaying content on digital signage devices. After I added that header, those pages would no longer load in an iframe on the digital signage devices’ browsers.

I thought I might be able to change SAMEORIGIN to ALLOW-FROM and list both the URI of my site and the URI of the digital signage page. However, the HTTP Header Field X-Frame-Options RFC indicates:

Wildcards or lists to declare multiple domains in one ALLOW-FROM statement are not permitted

The pages I wanted to exempt from the X-Frame-Options restriction exist in their own directory, /digitalsignage, so I tried to override the X-Frame-Options header in a .htaccess file:

Header always append X-Frame-Options ALLOW-ACCESS http://example.com

That caused a 500 Server Error. This message appeared in the error logs:

.htaccess: error: envclause should be in the form env=envar

The Header directive must be malformed, but I’m am not sure how. I did not determine how to properly format the statement so as not to produce that error, although several sites have pointed out that some browsers (Chrome, Safari) do not support ALLOW-ACCESS.

I changed the .htaccess file back to SAMEORIGIN, to match what was in the main site configuration:

Header always append X-Frame-Options SAMEORIGIN

I then noted that the response header sent by the server included SAMEORIGIN twice:

Header: SAMEORIGIN, SAMEORIGIN

That’s the expected behavior when using append. It appeared only once after I changed append to set:
Header always set X-Frame-Options SAMEORIGIN

I tried using set instead of append with ALLOW-ACCESS:

Header always set X-Frame-Options ALLOW-ACCESS http://example.com

But it still produced the same 500 Server Error.

After reading the documentation for Apache’s mod_headers, I realized that unset would allow me to remove the X-Frame-Options header from the /digitalsignage directory:
Header always unset X-Frame-Options

That worked, and the pages were successfully included as iframes in a page on the digital signage company’s site.

Canvas enrollments.csv and add_sis_stickiness

I have an enrollments.csv file for Instructure’s Canvas LMS, and I want all of the enrollments in it to “stick”–that is, to survive a batch mode SIS import. These are primarily course designers, and so they have no official standing in the class–and therefore are not in our database, and therefore are not included with regular updates to enrollments.

According to the Canvas documentation for SIS imports:

add_sis_stickiness – Boolean

This option, if present, will process all changes as if they were UI changes. This means that “stickiness” will be added to changed fields. This option is only processed if ‘override_sis_stickiness’ is also provided.

Source: https://canvas.instructure.com/doc/api/sis_imports.html#method.sis_imports_api.create

However, experience tells me otherwise. An inquiry to Instructure’s support confirms that add_sis_stickiness does not apply to enrollments. Enrollments added this way will be deleted following the next enrollments batch import.

The choices to preserve these course designer enrollments are basically to add each one manually using the web UI, or add them via the API. Either option will make the enrollments “stick.”

I opted to use the API. Since I already had a formatted input file, I wrote a short BASH script (with the help of several man pages and a couple StackOverflow pages) that reads the CSV and processes each row, adding the enrollment via the API:

headerrow=1
while read row; do
    if [ $headerrow -eq 0 ]
    then
        # get the SIS course ID
        cid="$(echo $row | cut -d',' -f1)"
        # get the SIS user ID
        uid="$(echo $row | cut -d',' -f2)"
        # get the role / enrollment type
        type="$(echo $row | cut -d',' -f3)"
        # reformat the enrollment type
        tid="$(echo $type | cut -c1 | tr [[:lower:]] [[:upper:]])""$(echo $type | cut -c2-)"Enrollment
        echo course is $cid
        echo user is $uid
        echo type is $tid
        result="$(curl https://[yourcanvassite].instructure.com/api/v1/courses/sis_course_id:$cid/enrollments -H 'Authorization: Bearer [REDACTED]' -X POST -F enrollment[type]=$tid -F enrollment[user_id]=sis_user_id:$uid -F 'enrollment[enrollment_state]=active' -F 'enrollment[notify]=false')"
        echo $result
    fi
    headerrow=0
done <enrollments.csv

Yammer Enterprise Block Users appears to be limited to 70 users per submission

This is just based on trial-and-error testing, but when I submit over 70 e-mail addresses at:

https://www.yammer.com/[my domain]/admin/blocked_email_addresses

I get one of the following errors:

  • Nginx 502 Bad Gateway
  • We’re sorry, but something went wrong.
    We’ve been notified about this issue and we’ll take a look at it shortly.

When I submit 70 addresses or fewer, it works as expected.

I have not yet discovered a way to block addresses via PowerShell.

Error retrieving Glacier vault inventory via AWS CLI: Unknown options: inventory-retrieval}’

Amazon’s Glacier service is great, but low-cost storage has other costs. For example, you might have a vault but not know what archives it contains. You can retrieve the vault inventory–essentially listing the contents of a directory–but like any Glacier retrieval, it may take several hours.

Using Downloading Vault Inventory using the REST API, I put this together:
C:\>aws glacier initiate-job --account-id - --vault-name my_poorly_named_vault --job-parameters '{"Type": "inventory-retrieval"}'

Which returned:

Unknown options: inventory-retrieval}'

I had taken the command directly from AWS’s example:
http://docs.aws.amazon.com/cli/latest/reference/glacier/initiate-job.html.

(Keep in mind that I had already followed the steps at Installing the Amazon Web Services Command Line Interface and Configuring the Amazon Web Services Command Line Interface.)

According to the documentation for the job-parameters option (http://docs.aws.amazon.com/sdkforruby/api/Aws/Glacier/Types/JobParameters.html#type-instance_method),
valid values are “archive-retrieval” and “inventory-retrieval”.

But the error message says inventory-retrieval}’. Why is it picking up the trailing curly brace and the apostrophe?

I formatted the job-parameters JSON in a file named aws-json.txt, with the curly braces on separate lines:

{
    "Type": "inventory-retrieval"
}

I tried this variation on the initiate-job command:
C:\>aws glacier initiate-job --account-id - --vault-name my_poorly_named_vault --job-parameters file://aws-json.txt

That worked!

The results returned:

{
    "jobId": "y8ugyoNzzusaf6Lv72G3hsjAA6O7nw5bJQ2u6J9TDnJ82_qx-lxnqrhSxIcGvOU1iiXoUhZboiojxsDu8gLQOfiJ7hR2",
    "location": "/123456789011/vaults/my_poorly_named_vault/jobs/y8ugyoNzzusaf6Lv72G3hsjAA6O7nw5bJQ2u6J9TDnJ82_qx-lxnqrhSxIcGvOU1iiXoUhZboiojxsDu8gLQOfiJ7hR2"
}

Online Advertising Click-Thru Rates, Revisited

A couple years ago, I wrote Online Advertisements and Statistical Analysis, in which I did my best to show that a past study of online advertising click-thru rates (CTRs) wasn’t worth the pixels it was printed on.

About a week ago, my wife and I were visiting friends, and I found myself in a room with 3 neuroscientists. The topic of statistics came up, and I managed to insert into conversation my small triumph in analyzing the click-thru study and determining both a confidence interval and the number of tests that would need to be run in order to have a meaningful confidence interval. “Sure,” one of the scientists says, “but what you should really do instead is a chi-square test for goodness-of-fit.”
Continue reading Online Advertising Click-Thru Rates, Revisited

Online Advertisements and Statistical Analysis

Quite a few years ago, I was in the online advertising business. My team and I created banner ads to run alongside web site content, to entice viewers to click on ads and find out more about advertiser offers. We scheduled ads to run alongside specific content. We targeted ads towards users in specific geographic regions, thanks to a browser cookie that told us their ZIP code. And we constantly managed inventory.

Although we made animated ads, we avoided anything that blinked. There were no monkeys to punch.

Click-Through Rate (or Click-Thru Rate or CTR) was a key measurement of an ad’s success. Although at first we would sometimes see click CTRs between 1-2% (meaning that an ad was clicked between 10 and 20 out of every 1000 views, or impressions), as online advertising proliferated, and as our systems got better at filtering out false impressions and clicks from various robots, crawlers, and spiders, CTRs trended much lower: 0.25% suddenly looked good, and 0.10% was not uncommon in some cases. That’s 1 click for every 1000 impressions.

That’s why we were insanely interested in a blog post we found, now presumably lost to the ages, that ran a set of 6 banner ads, which varied only slightly, and analyzed the results to determine what aspects of the ads could improve CTRs. Did including the phrase “click here” really help? If the words “click here” were in blue and underlined, like a typical web link, would that improve the CTR?
Continue reading Online Advertisements and Statistical Analysis