Category: Uncategorized

Jekyll quick start: Error installing jekyll, failed to build gem native extension

I’m looking at starting another blog at impractical.bot, as a home for my impractical chatbots. I thought I should check out a blog software other than WordPress, since all of my infosec colleagues make fun of me for using it. One option on my list is Jekyll, which builds static sites. That’s great if you are worried about resources (i.e. super-cheap hosting) and security.

The Jekyll site includes a 4-step quickstart. Unfortunately, it failed for me at step #1:

Error installing jekyll, failed to build gem native extension

I’m running this on a Fedora 27 virtual machine that I spun up for testing Jekyll:

$ cat /etc/system-release
Fedora release 27 (Twenty Seven)

My Ruby version is up-to-date:

$ ruby --version
ruby 2.4.3p205 (2017-12-14 revision 61247) [x86_64-linux]

Gem is installed and up-to-date:

$ gem --version
2.6.14.1

I took a closer look at the details of the error message:

gcc: error: /usr/lib/rpm/redhat/redhat-hardened-cc1: no such file or directory

I’m probably just missing a package, let’s see what package provides that file:

$ dnf provides redhat-hardened-cc1
Error: No Matches found

Fine, I let Google point me to a relevant StackOverflow post: G++ error:/usr/lib/rpm/redhat/redhat-hardened-cc1: No such file or directory

$ sudo dnf install redhat-rpm-config

I tried again:

$ sudo gem install bundler jekyll

It still failed, but with a different error:

make: g++: Command not found

I installed gcc-c++:

$ sudo dnf install gcc-c++

I tried to install Jekyll yet again:

$ sudo gem install bundler jekyll

That worked!

The following steps in Jekyll’s quickstart worked too:

$ jekyll new my-awesome-site
$ cd my-awesome-site
$ bundle exec jekyll serve

I copied my ~/my-awesome-site/_site folder to my document root (this can be automated with Jekyll, but I’m not there yet), and now, voila! my site is live:
http://impractical.bot

Analyzing text to find common terms using Python and NLTK

I just recently started playing with the Python NLTK (Natural Language ToolKit) to analyze text. The book Natural Language Processing with Python is available online and is very helpful if you’re just getting started.

At the beginning of the book the examples cover importing and analyzing text (primarily books) that you import from nltk (Getting Started with NLTK). It includes texts like Moby-Dick and Sense and Sensibility.

But you will probably want to analyze a source of your own. For example, I had text from a series of tweets debating political issues. The third chapter (Accessing Text from the Web and from Disk) has the answers:

First you need to turn raw text into tokens:

tokens = word_tokenize(raw)

Next turn your tokens into NLTK text:

text = nltk.Text(tokens)

Now you can treat it like the book examples in chapter 1.

I was analyzing a number number of tweets. One of the things I wanted to do was find common words in the tweets, to see if there were particular keywords that were common.

I was using the Python interpreter for my tests, and I did run into a couple errors with word_tokenize and later FreqDist, such as:

>>> fdist1 = FreqDist(text)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'FreqDist' is not defined

You can address this by importing the specific libraries:

>>> from nltk import FreqDist

Here are the commands, in order, that I ran to produce my list of common words — in this case, I was looking for words that appeared at least 3 times and that were at least 5 characters long:

>>> import nltk
>>> from nltk import word_tokenize
>>> from nltk import FreqDist

>>> with open("corpus-twitter", "r") as myfile:
...     raw = myfile.read().decode("utf8")

>>> tokens = word_tokenize(raw)
>>> text = nltk.Text(tokens)

>>> fdist = FreqDist(text)
>>> sorted(w for w in set(text) if len(w) >= 5 and fdist[w] >= 3)

[u'Americans', u'Detroit', u'Please', u'TaxReform', u'Thanks', u'There', u'Trump', u'about', u'against', u'always', u'anyone', u'argument', u'because', u'being', u'believe', u'context', u'could', u'debate', u'defend', u'diluted', u'dollars', u'enough', u'every', u'going', u'happened', u'heard', u'human', u'ideas', u'immigration', u'indefensible', u'logic', u'never', u'opinion', u'people', u'point', u'pragmatic', u'problem', u'problems', u'proposed', u'public', u'question', u'really', u'restricting', u'right', u'saying', u'school', u'scope', u'serious', u'should', u'solution', u'still', u'talking', u'their', u'there', u'think', u'thinking', u'thread', u'times', u'truth', u'trying', u'tweet', u'understand', u'until', u'welfare', u'where', u'world', u'would', u'wrong', u'years', u'yesterday']

It turns out the results weren’t as interesting as I’d hoped. A few interesting items–Detroit for example–but most of the words aren’t surprising given I was looking at tweets around political debate. Perhaps with a larger corpus there would be more stand-out words.

Guest SSID surprises on home wireless router

My current home Internet provider is CenturyLink, and with that I’m using their recommended Zyxel C1100Z “modem”.

Via the modem’s web interface you can configure up to 4 SSIDs. I have one set up for my devices with strong security settings, and another set up for guests with weaker security settings. One thing that surprised me: when I checked the list of attached devices, devices attached to the guest SSID were allocated IP addresses in the same address range as, and could communicate with, devices attached to my trusted home SSID.

The Zyxel C1100Z will let you create LAN subnets with different IP address ranges and settings, but a device on one subnet can still communicate with devices on another LAN subnet. This would let you at least configure a host firewall (on hosts that support a host firewall) to drop traffic from a particular address range (e.g. 192.168.100.0/24).

This is lunacy, though. Why would you create separate SSIDs with different security settings if the attached devices cannot be isolated from one another? I suspect that most users do not realize this. There are some settings you can change from one SSID to another, such as bandwidth throttling, but that seems like a secondary consideration to securing your network. Needless to say, my guest network has the same security settings as my trusted home network now.

I wondered if I had overlooked a setting somewhere, so I called to confirm with CenturyLink. The technician there was able to identify the SSIDs I had configured, suggesting that they have a backdoor into the modem they provided.

The moral of the story is: never use the equipment provided by your ISP.

Wifi on Raspberry Pi 3

I don’t run a GUI/Desktop on my Raspberry Pi devices. I don’t have monitors or keyboards connected to them — typically I log into them via SSH and manage them that way. I recently wanted to activate multiple network connections on my Raspberry Pi 3 Model B, so I decided to activate the wireless connection in addition to the wired connection that was already configured.

I tried following the steps at Setting WiFi up via the command line but got an error after the first step:

$ sudo iwlist wlan0 scan
wlan0     Interface doesn't support scanning : Network is down

I decided to check an see what devices were available via ifconfig:

$ ifconfig
eth0 ...
lo ...

No wireless device is listed there. I checked for all devices using the -a switch:

$ ifconfig -a
eth0 ...
lo ...
wlan0 ...

It’s there, wlan0, but it wasn’t active. I tried to bring the device up:

$ sudo ifconfig wlan0 up
SIOCSIFFLAGS: Operation not possible due to RF-kill

That error message is completely indecipherable to me! Fortunately, someone else had this error message too:
“SIOCSIFFLAGS: Operation not possible due to RF-kill”?

From that post I was able to determine that the wireless device was soft blocked:

$ sudo rfkill list
0: phy0: Wireless LAN
        Soft blocked: yes
        Hard blocked: no
1: hci0: Bluetooth
        Soft blocked: yes
        Hard blocked: no

How to unblock it? The following, described at the aforementioned post, worked. Although I’m not sure why I’m unblocking wifi instead of phy0:

$ sudo rfkill unblock wifi
$ sudo rfkill list
0: phy0: Wireless LAN
        Soft blocked: no
        Hard blocked: no
1: hci0: Bluetooth
        Soft blocked: yes
        Hard blocked: no

$ sudo ifconfig wlan0 up
$ ifconfig
eth0 ...
lo ...
wlan0 ...

The device is now active, but we still need to connect. First, I generated an encrypted passphrase:

$ wpa_passphrase "my_network_id" "my_network_password"
network={
        ssid="my_network_id"
        psk=8a9b456b28ef0707987622421592d3cc2fd22544ac281bced0f2028f4f4fcb85
}

Next, I appended that block of text to /etc/wpa_supplicant/wpa_supplicant.conf

Based on what I’ve read, that should be sufficient. The system should periodically detect changes to the wpa_supplicant.conf file and load the new settings automatically. I was impatient and rebooted. Apparently the following command should work too:

$ sudo wpa_cli reconfigure

After the reboot, I checked ifconfig for the wlan0 interface:

$ ifconfig wlan0 | grep 'inet addr'
          inet addr:192.168.0.23  Bcast:192.168.0.255  Mask:255.255.255.0

It has an address — success!

VMWare VSphere CLI vmware-cmd and the cfg parameter

I have a VMWare ESXi host. I can manage it through VMWare Fusion, although the options seem limited (I’m used to using vCenter Server, but I don’t have the license for that in this environment). I thought I’d give the VMWare vSphere Command Line Interface (CLI) a try. This was a mistake, but if you insist on following me down the same path, see the Drivers and Tools section on the VMWare vSphere Downloads page to get started.

First I tried vmware-cmd.

C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd
'vmware-cmd' is not recognized as an internal or external command,
operable program or batch file.

The actual file is vmware-cmd.pl (it’s in the bin folder).

I was able to run one command, to list the virtual machines on the host:
C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl -H esxi.osric.net -l
Enter username: chris
Enter password:

The documentation I was looking at was probably outdated, as the newer documentation gives better examples. But the version I was looking at indicated that most of the other commands require a <config_file_path> or <cfg> parameter. Unfortunately, it does not specify what those values consist of or what they might look like. There was a hint in the docs in vmware-cmd Overview:

vmware-cmd is a legacy tool and supports the usage of VMFS paths for virtual machine configuration files. As a rule, use datastore paths to access virtual machine configuration files.

It appears that <cfg> is the path to the VMX. There are several different ways to specify this:

Full path using GUID
C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl -H esxi.osric.net /vmfs/volumes/272c880d-a89548c1-a530-4bccbbad9507/benvolio/benvolio.vmx uptime
Enter username: chris
Enter password:
getuptime() = 7193

(The GUID is displayed in the output of the list of virtual machines from vmware-cmd.pl -l.)

Full path using Datastore Name
C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl -H esxi.osric.net "/vmfs/volumes/test vms/benvolio/benvolio.vmx" uptime
Enter username: chris
Enter password:
getuptime() = 7578

Datastore Name + relative path
C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl -H esxi.osric.net "[test vms] benvolio/benvolio.vmx" uptime
Enter username: chris
Enter password:
getuptime() = 7822

Entering my username and password every time is tedious though. According to the Connection Options for vmware-cmd:

The vmware-cmd vCLI command supports only a specific set of connection options. Other vCLI connection options are not supported, for example, you cannot use variables because the corresponding option is not supported.

In this case, I have the vSphere CLI installed on a password-protected Windows 2012r2 virtual machine, so I didn’t feel it was too much of a risk to set a temporary environment variable to store some of the connection options:

C:\Program Files (x86)\VMware\VMware vSphere CLI>SET VMOPTIONS=-H esxi.osric.net -U chris -P t0u6hpa55w0rd
C:\Program Files (x86)\VMware\VMware vSphere CLI>vmware-cmd.pl %VMOPTIONS% -l

Remember how the documentation said that “vmware-cmd is a legacy tool”?

I’m not sure what the official replacement is–possibly the PowerShell-based VMWare vSphere PowerCLI–but it turns out that the vSphere Client is free. Accessing your ESXi host via HTTPS should provide a link to download the installer. The vSphere Client does not appear to be something you can script against or automate, but for simple tasks it is much easier to use than vmware-cmd.pl.

Mail Users in Office 365 don’t have SMTP access

On-premises mail users (at least in Exchange 2010) had access to send mail as their organizational address through the on-premises SMTP server. However, mail users in Exchange Online cannot send mail as their organizational address using smtp.office365.com.

So what can we do?

In order to use SMTP, users need full mailboxes. But these users should not actually have mailbox access. As a test, created a mailbox and I disabled all email apps for the mailbox in Exchange Online:

An Office 365 user's email app settings
An Office 365 user’s email app settings

The test user was no longer able to log in to Outlook on the Web (also known as OWA). SMTP still worked. Email forwarding still worked (although the user would not be able to set the forwarding address themselves).

Creating a user mailbox requires a user license, whereas mail users do not require a license. If you have a lot of on-premises mail users that now need full mailboxes, this could be problematic.

Summary:
In Exchange Online, a UserMailbox with all email apps disabled is equivalent to an on-premises Exchange MailUser, except that the former requires a license.

Middleman error – `block in replace_gem’: middleman-cli is not part of the bundle. Add it to Gemfile.

I ran into an error while setting up a Middleman site on my PC. I already had Ruby and RubyGems installed, so I followed the instructions to install Middleman and start a new Middleman site:

gem install middleman
middleman init project
cd project

The next step was to start the preview web server, which produced an error:

$ bundle exec middleman server
DL is deprecated, please use Fiddle
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/bundler-1.7.7/lib/bundler/r
ubygems_integration.rb:256:in `block in replace_gem': middleman-cli is not part
of the bundle. Add it to Gemfile. (Gem::LoadError)
from c:/RailsInstaller/Ruby2.1.0/bin/middleman:22:in `<main>'</main>

Continue reading Middleman error – `block in replace_gem’: middleman-cli is not part of the bundle. Add it to Gemfile.

Downgrading a Debian package

After I updated a Debian Wheezy server to Debian Jessie, I was having some problems with VLC. I was using the cvlc to capture and record video streams, but the capture would fail after 2 seconds. A colleague suggested that I leave the OS version alone, but downgrade VLC to confirm that the new version of VLC was causing the problem. A sound idea, but how do you downgrade a Debian package?

I found a variety of helpful sites and came up with the following:

  1. Add the source for the downgraded package the apt config
  2. Specify the target release in the apt config
  3. Use “pinning” to tell apt to use the older versions for the package and its dependencies
  4. Use apt-get install to install the dependencies and the package

Details as follows:

Add the source for the downgraded package the apt config
In this case, I needed to add the
I left my /etc/apt/sources.list as-is and added the following to /etc/apt/sources.list.d/vlc.list:

deb http://mirror.cc.columbia.edu/debian/ wheezy main non-free contrib
deb-src http://mirror.cc.columbia.edu/debian/ wheezy main non-free contrib

deb http://security.debian.org/ wheezy/updates main contrib non-free
deb-src http://security.debian.org/ wheezy/updates main contrib non-free

deb http://mirror.cc.columbia.edu/debian/ wheezy-updates main contrib non-free
deb-src http://mirror.cc.columbia.edu/debian/ wheezy-updates main contrib non-free

I used Columbia’s mirror because it’s fast and geographically nearer than most of the other mirrors. I’m sure not all of those sources were necessary (there are no security-related packages in VLC) but it didn’t hurt anything to include them.

Specify the target release in the apt config
I added the following to /etc/apt/apt.conf.d/80targetrelease to specify that I want Jessie to be the target/default release:

APT::Default-Release "jessie";

Use “pinning” to tell apt to use the older versions for package and its dependencies
I created the file /etc/apt/preferences.d/vlc containing the following. I added to the list of packages as apt-get install failed due to missing dependencies. The pin-priority 1001 was suggested by several sites to be used only in the case of downgrading a package.

Package: vlc
Pin: release n=wheezy
Pin-Priority: 1001

Package: vlc-data
Pin: release n=wheezy
Pin-Priority: 1001

Package: vlc-nox
Pin: release n=wheezy
Pin-Priority: 1001

Package: libvlc5
Pin: release n=wheezy
Pin-Priority: 1001

Package: libvlccore5
Pin: release n=wheezy
Pin-Priority: 1001

Package: vlc-plugin-notify
Pin: release n=wheezy
Pin-Priority: 1001

Package: vlc-plugin-pulse
Pin: release n=wheezy
Pin-Priority: 1001

Package: xdg-utils
Pin: release n=wheezy
Pin-Priority: 1001

Use apt-get install to install the dependencies and the package
I started by trying to apt-get install vlc but it complained about missing dependencies. I added the dependencies to the preferences file as describe in the section above, and then was able to install the following packages from Debian Wheezy:

apt-get install libvlccore5
apt-get install vlc-data
apt-get install libvlc5
apt-get install vlc-nox

Alternatively, I believe I could have skipped the preferences file and setting the pin-priorities by specifying the target Debian version when running apt-get install:

apt-get install -t wheezy libvlccore5
apt-get install -t wheezy vlc-data
apt-get install -t wheezy libvlc5
apt-get install -t wheezy vlc-nox

However, I’m not sure that the desired package version would be preserved after running apt-get dist-upgrade.

References
The following sites were helpful to me while I was figuring out how to do this, and if you are interested in pin-priority and the different values to use in different scenarios, I definitely recommend the first link:

Applying per directory X-Frame-Options headers in Apache

To help prevent against click-jacking, I had applied the following to my Apache 2.2 configuration based on the suggestions described in OWASP’s Clickjacking Defense Cheat Sheet and Mozilla Developer Network’s The X-Frame-Options response header:

Header always append X-Frame-Options SAMEORIGIN

However, my site has certain pages that are included in an iframe on another site, for the purpose of displaying content on digital signage devices. After I added that header, those pages would no longer load in an iframe on the digital signage devices’ browsers.

I thought I might be able to change SAMEORIGIN to ALLOW-FROM and list both the URI of my site and the URI of the digital signage page. However, the HTTP Header Field X-Frame-Options RFC indicates:

Wildcards or lists to declare multiple domains in one ALLOW-FROM statement are not permitted

The pages I wanted to exempt from the X-Frame-Options restriction exist in their own directory, /digitalsignage, so I tried to override the X-Frame-Options header in a .htaccess file:

Header always append X-Frame-Options ALLOW-ACCESS http://example.com

That caused a 500 Server Error. This message appeared in the error logs:

.htaccess: error: envclause should be in the form env=envar

The Header directive must be malformed, but I’m am not sure how. I did not determine how to properly format the statement so as not to produce that error, although several sites have pointed out that some browsers (Chrome, Safari) do not support ALLOW-ACCESS.

I changed the .htaccess file back to SAMEORIGIN, to match what was in the main site configuration:

Header always append X-Frame-Options SAMEORIGIN

I then noted that the response header sent by the server included SAMEORIGIN twice:

Header: SAMEORIGIN, SAMEORIGIN

That’s the expected behavior when using append. It appeared only once after I changed append to set:
Header always set X-Frame-Options SAMEORIGIN

I tried using set instead of append with ALLOW-ACCESS:

Header always set X-Frame-Options ALLOW-ACCESS http://example.com

But it still produced the same 500 Server Error.

After reading the documentation for Apache’s mod_headers, I realized that unset would allow me to remove the X-Frame-Options header from the /digitalsignage directory:
Header always unset X-Frame-Options

That worked, and the pages were successfully included as iframes in a page on the digital signage company’s site.

Canvas enrollments.csv and add_sis_stickiness

I have an enrollments.csv file for Instructure’s Canvas LMS, and I want all of the enrollments in it to “stick”–that is, to survive a batch mode SIS import. These are primarily course designers, and so they have no official standing in the class–and therefore are not in our database, and therefore are not included with regular updates to enrollments.

According to the Canvas documentation for SIS imports:

add_sis_stickiness – Boolean

This option, if present, will process all changes as if they were UI changes. This means that “stickiness” will be added to changed fields. This option is only processed if ‘override_sis_stickiness’ is also provided.

Source: https://canvas.instructure.com/doc/api/sis_imports.html#method.sis_imports_api.create

However, experience tells me otherwise. An inquiry to Instructure’s support confirms that add_sis_stickiness does not apply to enrollments. Enrollments added this way will be deleted following the next enrollments batch import.

The choices to preserve these course designer enrollments are basically to add each one manually using the web UI, or add them via the API. Either option will make the enrollments “stick.”

I opted to use the API. Since I already had a formatted input file, I wrote a short BASH script (with the help of several man pages and a couple StackOverflow pages) that reads the CSV and processes each row, adding the enrollment via the API:

headerrow=1
while read row; do
    if [ $headerrow -eq 0 ]
    then
        # get the SIS course ID
        cid="$(echo $row | cut -d',' -f1)"
        # get the SIS user ID
        uid="$(echo $row | cut -d',' -f2)"
        # get the role / enrollment type
        type="$(echo $row | cut -d',' -f3)"
        # reformat the enrollment type
        tid="$(echo $type | cut -c1 | tr [[:lower:]] [[:upper:]])""$(echo $type | cut -c2-)"Enrollment
        echo course is $cid
        echo user is $uid
        echo type is $tid
        result="$(curl https://[yourcanvassite].instructure.com/api/v1/courses/sis_course_id:$cid/enrollments -H 'Authorization: Bearer [REDACTED]' -X POST -F enrollment[type]=$tid -F enrollment[user_id]=sis_user_id:$uid -F 'enrollment[enrollment_state]=active' -F 'enrollment[notify]=false')"
        echo $result
    fi
    headerrow=0
done <enrollments.csv