Removing exceptions from a list using Bash (with sed and awk)

  • I have a CSV file, a list of 1000+ users and user properties.
  • I have a list of exceptions (users to be excluded from processing), one user per line, about 50 total.

How can I remove the exceptions from the list?

# make a copy of the original list
cp list-of-1000.csv list-of-1000-less-exceptions.csv
# loop through each line in exceptions.txt and remove matching lines from the copy
while read line; do sed -i "/${line}/d" list-of-1000-less-exceptions.csv; done < exceptions.txt

This is a little simplistic and could be a problem if any usernames are subsets of other usernames. (For example, if user ‘bob’ is on the list of exceptions, but the list of users also contains ‘bobb’, both would be deleted.)

In the particular instance I am dealing with, the username is conveniently the first field in the CSV file. This allows me to match the start of the line and the comma following the username:

while read line; do sed -i "/^${line},/d" list-of-1000-less-exceptions.csv; done < exceptions.txt

What if the username was the third field in the CSV instead of the first?

Use awk:
awk -F, -vOFS=, '{print $3,$0}' list-of-exceptions.csv > copy-of-list-of-exceptions.csv

  • -F, sets the field separator to a comma (defaults to whitespace)
  • -vOFS=, sets the Output Field Separator (OFS) to a comma (defaults to a space)
  • $3 prints the third field
  • $0 prints all the fields, with the specified field separator between them

while read line; do sed -i "/^${line},/d" copy-of-1000-less-exceptions.csv; done < exceptions.txt

Now there’s still an extra username in this file. Maybe that doesn’t matter, but maybe it does. There are several ways to remove it–here’s one:

awk -F, -vOFS=, '$1=""; print $0' copy-of-1000-less-exceptions.csv | sed 's/^,//' > list-of-1000-less-exceptions.csv

  • -F, sets the field separator to a comma (defaults to whitespace)
  • -vOFS=, sets the Output Field Separator (OFS) to a comma (defaults to a space)
  • $1="" sets the first field to an empty string
  • print $0 prints all the fields

The result of the awk command has an initial comma on each line. The first field is still there, it’s just set to an empty string. I used sed to remove it.

You could also use sed alone to remove the extra username field:
sed -i 's/^[^,]*,//' copy-of-1000-less-exceptions.csv

Middleman error – `block in replace_gem’: middleman-cli is not part of the bundle. Add it to Gemfile.

I ran into an error while setting up a Middleman site on my PC. I already had Ruby and RubyGems installed, so I followed the instructions to install Middleman and start a new Middleman site:

gem install middleman
middleman init project
cd project

The next step was to start the preview web server, which produced an error:

$ bundle exec middleman server
DL is deprecated, please use Fiddle
c:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/bundler-1.7.7/lib/bundler/r
ubygems_integration.rb:256:in `block in replace_gem': middleman-cli is not part
of the bundle. Add it to Gemfile. (Gem::LoadError)
from c:/RailsInstaller/Ruby2.1.0/bin/middleman:22:in `<main>'</main>

Continue reading Middleman error – `block in replace_gem’: middleman-cli is not part of the bundle. Add it to Gemfile.

Node.js – Error ‘console’ is undefined

I just installed Node.js on a new system and wanted to make sure it worked. I tested it with a simple hello world script.

Contents of the node.js file
console.log("Hello node");

Then I attempted to run it:
C:\>node node.js

Which produced the following error:
Script: C:\node.js
Line: 1
Char: 1
Error: 'console' is undefined
Code: 800A1391
Source: Microsoft JScript runtime error

Weird. And Microsoft JScript runtime? Very weird.

Turns out, Windows is trying to run the Javascript, node.js, not via Node.js but natively in Windows.

If I rename the file hellonode.js I can reproduce the same error:
C:\>hellonode

Or I can run the intended file via Node.js:
C:\>node hellonode.js
Hello node

I did not realize that the Windows command prompt could/would execute Javascript files natively.

Pipe array to Powershell script

I created a Powershell script to set a few mailbox properties. I wanted to pipe in an array of mailbox objects, i.e. the results of a Get-Mailbox command, like so:

$Mailboxes | C:\Set-MailboxProperties.ps1

However, Set-MailboxProperties.ps1 only processed the first item in the array.

Via Google I found this helpful StackOverflow post, How to pass an array as a parameter to another script? Although the answers did not include an example exactly like mine, their solution works just the same:

(,$Mailboxes) | C:\Set-MailboxProperties.ps1

PowerShell Ellipsis (dot dot dot)

Sometimes when you retrieve an object via PowerShell, some of the properties are truncated, denoted by an ellipsis (“…”).

For example:
Get-Mailbox chris | Select AddressListMembership

AddressListMembership
---------------------
{\Staff Global Address List, \Staff, \IT Staff, \Exchange Admins...}

How do you see the full list? There are a couple ways:

Select -ExpandProperty
Get-Mailbox chris | Select -ExpandProperty AddressListMembership

$FormatEnumerationLimit =-1
This is a per-session variable in PowerShell. By default the value is 4, but if you change it to -1 it will enumerate all items. This will affect every property of every object, so it may be more than you need.

Free/Busy Time Segmentation in Exchange Online

By default, all users in the same Exchange Online environment can view each other’s free/busy time. Using the Organization–Sharing settings you can share more information, but not less.

Exchange Online Sharing Rule
Unchecking the ‘Share your calendar folder’ box does not turn off calendar sharing. Counterintuitive!

Individuals can adjust their own free/busy time sharing in Outlook or Outlook Web App (OWA). But what if you have less-privileged users who should not be able to view another user’s free/busy time, for example, temporary employees or contract workers? Can they be restricted from viewing calendar information for other users?

It can be done, but it’s not simple.

My 3-part approach, summarized:

  1. Change each user’s sharing settings for the Default user to None via PowerShell
  2. Create a mail-enabled universal security group containing all privileged users. (Fortunately, this group already existed within my organization.)
  3. Change each user’s sharing settings for the security group created above to AvailabilityOnly via PowerShell (to allow just Free/Busy visibility)

I found Add Calendar Permissions in Office 365 via Powershell, which was a tremendous help in discovering the format of the calendar folder. For example, to adjust the Default user’s access to chris@example.com’s calendar to None, use the following PowerShell command:
Set-MailboxFolderPermission -Identity chris@example.com:\calendar -user Default -AccessRights None

Then I tried to add permissions for the security group:
$mycal = 'chris@example.com:\calendar'
Set-MailboxFolderPermission -Identity $mycal -User privileged-users-security-group@example.com -AccessRights AvailabilityOnly

Error:
There is no existing permission entry found for user: privileged-users-security-group.
+ CategoryInfo : NotSpecified: (:) [Set-MailboxFolderPermission], UserNotFoundInPermissionEntryException
+ FullyQualifiedErrorId : [Server=BLUPR0101MB1603,RequestId=d057882d-5663-417d-a614-ce73e5ab0565,TimeStamp=3/15/20
16 3:41:20 PM] [FailureCategory=Cmdlet-UserNotFoundInPermissionEntryException] B65CA2A0,Microsoft.Exchange.Managem
ent.StoreTasks.SetMailboxFolderPermission
+ PSComputerName : ps.outlook.com

Thanks to Setup secretary permissions to manage Calendar in Office 365, I discovered that the above error occurred because the security group had no current settings for the specified calendar. In that case, the Add-MailboxFolderPermission is the appropriate command:

Add-MailboxFolderPermission -Identity $mycal -User privileged-users-security-group@example.com -AccessRights AvailabilityOnly

Before running this across all of our users, I wanted to find out which users had customized their free/busy sharing settings. If they had customized them, I wanted to preserve their settings. For example, I decided to get the Default user sharing settings for the sales department users’ calendars:

$DeptMailboxes = Get-Mailbox -Filter {CustomAttribute2 -eq 'sales'}
ForEach ($Mailbox In $DeptMailboxes) { $Calendar = $Mailbox.UserPrincipalName + ":\calendar"; Get-MailboxFolderPermission -Identity $Calendar -User Default}

Unfortunately, the above did not return all of the properties needed to identify the calendars in question:
Calendar Default {AvailabilityOnly}
Calendar Default {LimitedDetails}
Calendar Default {AvailabilityOnly}
Calendar Default {AvailabilityOnly}

I specified a list of properties that was more useful:
ForEach ($Mailbox In $DeptMailboxes) { $Calendar = $Mailbox.UserPrincipalName + ":\calendar"; Get-MailboxFolderPermission -Identity $Calendar -User Default | Select Identity,FolderName,User,AccessRights }

Fortunately, only a handful of the users in my organization had customized their sharing settings, so I simply noted their settings and re-applied them after running these settings across all users in the organization:

$AllMailboxes = Get-Mailbox
ForEach ($Mailbox In $AllMailboxes) { $Calendar = $Mailbox.UserPrincipalName + ":\calendar"; Set-MailboxFolderPermission -Identity $Calendar -User Default -AccessRights None; Add-MailboxFolderPermission -Identity $Calendar -User privileged-users-security-group@example.com -AccessRights AvailabilityOnly }

This achieved the desired free/busy time segmentation. However, there’s one snag: what happens when new users are added? They will have the default sharing settings. That means that every time a new user is added, these steps will need to be run for that new user. I created the following PowerShell script — I can pipe the results of Get-Mailbox to this script to apply the customizations described above:

param(  
    [Parameter(
        Position=0, 
        Mandatory=$true, 
        ValueFromPipeline=$true,
        ValueFromPipelineByPropertyName=$true)
    ]
    [Object[]]$Mailbox
)

Process {
    $Calendar = $Mailbox.UserPrincipalName + ":\calendar"
    Set-MailboxFolderPermission -Identity $Calendar -User Default -AccessRights None
    Add-MailboxFolderPermission -Identity $Calendar -User 'privileged-users-security-group@example.com' -AccessRights AvailabilityOnly
}

To run the script (assuming it is named Set-CustomFreeBusySharing.ps1):
Get-Mailbox -Identity bob@example.com | ./Set-CustomFreeBusySharing.ps1

Fully integrating that into my account creation process is a job for another day.

One other thing to note: users can still choose to modify their free/busy sharing with the Default user, in case they do want/need to share their availability with all users in the organization.

Other sites that had useful information while I researched this issue:

Set-MsolUserLicense : A parameter cannot be found that matches parameter name ‘UsageLocation’

I wanted to apply an Office 365 (O365) license to a user via PowerShell:

PowerShell Command
Set-MsolUserLicense -UserPrincipalName "johndoe@example.com" -AddLicenses "exampletenant:EXCHANGESTANDARD_ALUMNI"

Error
Set-MsolUserLicense : You must provide a required property: Parameter name: UsageLocation

Fine. What is this UsageLocation parameter? See Assign licenses to user accounts with Office 365 PowerShell:

You can assign licenses only to user accounts that have the UsageLocation property set to a valid ISO 3166-1 alpha-2 country code. For example, US for the United States, and FR for France.

OK, so US for United States.

PowerShell Command
Set-MsolUserLicense -UserPrincipalName "johndoe@example.com" -AddLicenses "exampletenant:EXCHANGESTANDARD_ALUMNI" -UsageLocation US

Error
Set-MsolUserLicense : A parameter cannot be found that matches parameter name 'UsageLocation'.

Really, Microsoft? Is it a required parameter or is it not a parameter? Make up your minds!

Turns out, it’s a parameter of the Set-MsolUser cmdlet:

PowerShell Commands
Set-MsolUser -UserPrincipalName "johndoe@example.com" -UsageLocation US
Set-MsolUserLicense -UserPrincipalName "johndoe@example.com" -AddLicenses "exampletenant:EXCHANGESTANDARD_ALUMNI"

The license was applied successfully.

Querying an Oracle database from Powershell

I needed to query Oracle for information to use in a Powershell script. I found the following comprehensive blog post with details on how to do so using ODP.NET: Use Oracle ODP.NET and PowerShell to Simplify Data Access

The article is extremely long and goes into a lot of depth. The following is a short summary of the first steps, with just enough to get started:

  1. Download Oracle Data Provider for .NET (ODP.NET). (If that link doesn’t work just search for “Oracle ODP.NET”.)
    • Select “Download the latest ODP.NET production release.”
    • Select “64-bit ODAC Downloads”
    • Select “ODP.NET_Managed_ODAC12cR4.zip”
  2. Extract the ZIP file to C:\, which creates C:\ODP.NET_Managed_ODAC12cR4.
  3. Run cmd as administrator, navigate to C:\ODP.NET_Managed_ODAC12cR4, and run:
    install_odpm.bat C:\oracle\instantclient_10_2 both

In Powershell, add the DLL and set up a database connection and a query:

Add-Type -Path "C:\Users\chris\ODP.NET_Managed_ODAC12cR4\odp.net\managed\common\Oracle.ManagedDataAccess.dll"
$username = Read-Host -Prompt "Enter database username"
$password = Read-Host -Prompt "Enter database password"
$datasource = Read-Host -Prompt "Enter database TNS name"
$query = "SELECT first_name, last_name FROM users.user WHERE first_name = 'Chris' ORDER BY last_name"
$connectionString = 'User Id=' + $username + ';Password=' + $password + ';Data Source=' + $datasource
$connection = New-Object Oracle.ManagedDataAccess.Client.OracleConnection($connectionString)
$connection.open()
$command=$connection.CreateCommand()
$command.CommandText=$query
$reader=$command.ExecuteReader()
while ($reader.Read()) {
$reader.GetString(1) + ', ' + $reader.GetString(0)
}
$connection.Close()

Output of the above:

Brown, Chris
Carter, Chris
Jones, Chris
Smith, Chris

Downgrading a Debian package

After I updated a Debian Wheezy server to Debian Jessie, I was having some problems with VLC. I was using the cvlc to capture and record video streams, but the capture would fail after 2 seconds. A colleague suggested that I leave the OS version alone, but downgrade VLC to confirm that the new version of VLC was causing the problem. A sound idea, but how do you downgrade a Debian package?

I found a variety of helpful sites and came up with the following:

  1. Add the source for the downgraded package the apt config
  2. Specify the target release in the apt config
  3. Use “pinning” to tell apt to use the older versions for the package and its dependencies
  4. Use apt-get install to install the dependencies and the package

Details as follows:

Add the source for the downgraded package the apt config
In this case, I needed to add the
I left my /etc/apt/sources.list as-is and added the following to /etc/apt/sources.list.d/vlc.list:

deb http://mirror.cc.columbia.edu/debian/ wheezy main non-free contrib
deb-src http://mirror.cc.columbia.edu/debian/ wheezy main non-free contrib

deb http://security.debian.org/ wheezy/updates main contrib non-free
deb-src http://security.debian.org/ wheezy/updates main contrib non-free

deb http://mirror.cc.columbia.edu/debian/ wheezy-updates main contrib non-free
deb-src http://mirror.cc.columbia.edu/debian/ wheezy-updates main contrib non-free

I used Columbia’s mirror because it’s fast and geographically nearer than most of the other mirrors. I’m sure not all of those sources were necessary (there are no security-related packages in VLC) but it didn’t hurt anything to include them.

Specify the target release in the apt config
I added the following to /etc/apt/apt.conf.d/80targetrelease to specify that I want Jessie to be the target/default release:

APT::Default-Release "jessie";

Use “pinning” to tell apt to use the older versions for package and its dependencies
I created the file /etc/apt/preferences.d/vlc containing the following. I added to the list of packages as apt-get install failed due to missing dependencies. The pin-priority 1001 was suggested by several sites to be used only in the case of downgrading a package.

Package: vlc
Pin: release n=wheezy
Pin-Priority: 1001

Package: vlc-data
Pin: release n=wheezy
Pin-Priority: 1001

Package: vlc-nox
Pin: release n=wheezy
Pin-Priority: 1001

Package: libvlc5
Pin: release n=wheezy
Pin-Priority: 1001

Package: libvlccore5
Pin: release n=wheezy
Pin-Priority: 1001

Package: vlc-plugin-notify
Pin: release n=wheezy
Pin-Priority: 1001

Package: vlc-plugin-pulse
Pin: release n=wheezy
Pin-Priority: 1001

Package: xdg-utils
Pin: release n=wheezy
Pin-Priority: 1001

Use apt-get install to install the dependencies and the package
I started by trying to apt-get install vlc but it complained about missing dependencies. I added the dependencies to the preferences file as describe in the section above, and then was able to install the following packages from Debian Wheezy:

apt-get install libvlccore5
apt-get install vlc-data
apt-get install libvlc5
apt-get install vlc-nox

Alternatively, I believe I could have skipped the preferences file and setting the pin-priorities by specifying the target Debian version when running apt-get install:

apt-get install -t wheezy libvlccore5
apt-get install -t wheezy vlc-data
apt-get install -t wheezy libvlc5
apt-get install -t wheezy vlc-nox

However, I’m not sure that the desired package version would be preserved after running apt-get dist-upgrade.

References
The following sites were helpful to me while I was figuring out how to do this, and if you are interested in pin-priority and the different values to use in different scenarios, I definitely recommend the first link:

Applying per directory X-Frame-Options headers in Apache

To help prevent against click-jacking, I had applied the following to my Apache 2.2 configuration based on the suggestions described in OWASP’s Clickjacking Defense Cheat Sheet and Mozilla Developer Network’s The X-Frame-Options response header:

Header always append X-Frame-Options SAMEORIGIN

However, my site has certain pages that are included in an iframe on another site, for the purpose of displaying content on digital signage devices. After I added that header, those pages would no longer load in an iframe on the digital signage devices’ browsers.

I thought I might be able to change SAMEORIGIN to ALLOW-FROM and list both the URI of my site and the URI of the digital signage page. However, the HTTP Header Field X-Frame-Options RFC indicates:

Wildcards or lists to declare multiple domains in one ALLOW-FROM statement are not permitted

The pages I wanted to exempt from the X-Frame-Options restriction exist in their own directory, /digitalsignage, so I tried to override the X-Frame-Options header in a .htaccess file:

Header always append X-Frame-Options ALLOW-ACCESS http://example.com

That caused a 500 Server Error. This message appeared in the error logs:

.htaccess: error: envclause should be in the form env=envar

The Header directive must be malformed, but I’m am not sure how. I did not determine how to properly format the statement so as not to produce that error, although several sites have pointed out that some browsers (Chrome, Safari) do not support ALLOW-ACCESS.

I changed the .htaccess file back to SAMEORIGIN, to match what was in the main site configuration:

Header always append X-Frame-Options SAMEORIGIN

I then noted that the response header sent by the server included SAMEORIGIN twice:

Header: SAMEORIGIN, SAMEORIGIN

That’s the expected behavior when using append. It appeared only once after I changed append to set:
Header always set X-Frame-Options SAMEORIGIN

I tried using set instead of append with ALLOW-ACCESS:

Header always set X-Frame-Options ALLOW-ACCESS http://example.com

But it still produced the same 500 Server Error.

After reading the documentation for Apache’s mod_headers, I realized that unset would allow me to remove the X-Frame-Options header from the /digitalsignage directory:
Header always unset X-Frame-Options

That worked, and the pages were successfully included as iframes in a page on the digital signage company’s site.