PowerShell Ellipsis (dot dot dot)

Sometimes when you retrieve an object via PowerShell, some of the properties are truncated, denoted by an ellipsis (“…”).

For example:
Get-Mailbox chris | Select AddressListMembership

AddressListMembership
---------------------
{\Staff Global Address List, \Staff, \IT Staff, \Exchange Admins...}

How do you see the full list? There are a couple ways:

Select -ExpandProperty
Get-Mailbox chris | Select -ExpandProperty AddressListMembership

$FormatEnumerationLimit =-1
This is a per-session variable in PowerShell. By default the value is 4, but if you change it to -1 it will enumerate all items. This will affect every property of every object, so it may be more than you need.

Tags: , ,

Wednesday, April 6th, 2016 Powershell, Tips & Tricks No Comments

Free/Busy Time Segmentation in Exchange Online

By default, all users in the same Exchange Online environment can view each other’s free/busy time. Using the Organization–Sharing settings you can share more information, but not less.

Exchange Online Sharing Rule

Unchecking the ‘Share your calendar folder’ box does turn off calendar sharing. Counterintuitive!

Individuals can adjust their own free/busy time sharing in Outlook or Outlook Web App (OWA). But what if you have less-privileged users who should not be able to view another user’s free/busy time, for example, temporary employees or contract workers? Can they be restricted from viewing calendar information for other users?

It can be done, but it’s not simple.

My 3-part approach, summarized:

  1. Change each user’s sharing settings for the Default user to None via PowerShell
  2. Create a mail-enabled universal security group containing all privileged users. (Fortunately, this group already existed within my organization.)
  3. Change each user’s sharing settings for the security group created above to AvailabilityOnly via PowerShell (to allow just Free/Busy visibility)

I found Add Calendar Permissions in Office 365 via Powershell, which was a tremendous help in discovering the format of the calendar folder. For example, to adjust the Default user’s access to chris@example.com’s calendar to None, use the following PowerShell command:
Set-MailboxFolderPermission -Identity chris@example.com:\calendar -user Default -AccessRights None

Then I tried to add permissions for the security group:
$mycal = 'chris@example.com:\calendar'
Set-MailboxFolderPermission -Identity $mycal -User privileged-users-security-group@example.com -AccessRights AvailabilityOnly

Error:
There is no existing permission entry found for user: privileged-users-security-group.
+ CategoryInfo : NotSpecified: (:) [Set-MailboxFolderPermission], UserNotFoundInPermissionEntryException
+ FullyQualifiedErrorId : [Server=BLUPR0101MB1603,RequestId=d057882d-5663-417d-a614-ce73e5ab0565,TimeStamp=3/15/20
16 3:41:20 PM] [FailureCategory=Cmdlet-UserNotFoundInPermissionEntryException] B65CA2A0,Microsoft.Exchange.Managem
ent.StoreTasks.SetMailboxFolderPermission
+ PSComputerName : ps.outlook.com

Thanks to Setup secretary permissions to manage Calendar in Office 365, I discovered that the above error occurred because the security group had no current settings for the specified calendar. In that case, the Add-MailboxFolderPermission is the appropriate command:

Add-MailboxFolderPermission -Identity $mycal -User privileged-users-security-group@example.com -AccessRights AvailabilityOnly

Before running this across all of our users, I wanted to find out which users had customized their free/busy sharing settings. If they had customized them, I wanted to preserve their settings. For example, I decided to get the Default user sharing settings for the sales department users’ calendars:

$DeptMailboxes = Get-Mailbox -Filter {CustomAttribute2 -eq 'sales'}
ForEach ($Mailbox In $DeptMailboxes) { $Calendar = $Mailbox.UserPrincipalName + ":\calendar"; Get-MailboxFolderPermission -Identity $Calendar -User Default}

Unfortunately, the above did not return all of the properties needed to identify the calendars in question:
Calendar Default {AvailabilityOnly}
Calendar Default {LimitedDetails}
Calendar Default {AvailabilityOnly}
Calendar Default {AvailabilityOnly}

I specified a list of properties that was more useful:
ForEach ($Mailbox In $DeptMailboxes) { $Calendar = $Mailbox.UserPrincipalName + ":\calendar"; Get-MailboxFolderPermission -Identity $Calendar -User Default | Select Identity,FolderName,User,AccessRights }

Fortunately, only a handful of the users in my organization had customized their sharing settings, so I simply noted their settings and re-applied them after running these settings across all users in the organization:

$AllMailboxes = Get-Mailbox
ForEach ($Mailbox In $AllMailboxes) { $Calendar = $Mailbox.UserPrincipalName + ":\calendar"; Set-MailboxFolderPermission -Identity $Calendar -User Default -AccessRights None; Add-MailboxFolderPermission -Identity $Calendar -User privileged-users-security-group@example.com -AccessRights AvailabilityOnly }

This achieved the desired free/busy time segmentation. However, there’s one snag: what happens when new users are added? They will have the default sharing settings. That means that every time a new user is added, these steps will need to be run for that new user. I created the following PowerShell script — I can pipe the results of Get-Mailbox to this script to apply the customizations described above:

param(  
    [Parameter(
        Position=0, 
        Mandatory=$true, 
        ValueFromPipeline=$true,
        ValueFromPipelineByPropertyName=$true)
    ]
    [Object[]]$Mailbox
)

Process {
    $Calendar = $Mailbox.UserPrincipalName + ":\calendar"
    Set-MailboxFolderPermission -Identity $Calendar -User Default -AccessRights None
    Add-MailboxFolderPermission -Identity $Calendar -User 'privileged-users-security-group@example.com' -AccessRights AvailabilityOnly
}

To run the script (assuming it is named Set-CustomFreeBusySharing.ps1):
Get-Mailbox -Identity bob@example.com | ./Set-CustomFreeBusySharing.ps1

Fully integrating that into my account creation process is a job for another day.

One other thing to note: users can still choose to modify their free/busy sharing with the Default user, in case they do want/need to share their availability with all users in the organization.

Other sites that had useful information while I researched this issue:

Tags: , , , , , ,

Tuesday, March 29th, 2016 Powershell No Comments

Set-MsolUserLicense : A parameter cannot be found that matches parameter name ‘UsageLocation’

I wanted to apply an Office 365 (O365) license to a user via PowerShell:

PowerShell Command
Set-MsolUserLicense -UserPrincipalName "johndoe@example.com" -AddLicenses "exampletenant:EXCHANGESTANDARD_ALUMNI"

Error
Set-MsolUserLicense : You must provide a required property: Parameter name: UsageLocation

Fine. What is this UsageLocation parameter? See Assign licenses to user accounts with Office 365 PowerShell:

You can assign licenses only to user accounts that have the UsageLocation property set to a valid ISO 3166-1 alpha-2 country code. For example, US for the United States, and FR for France.

OK, so US for United States.

PowerShell Command
Set-MsolUserLicense -UserPrincipalName "johndoe@example.com" -AddLicenses "exampletenant:EXCHANGESTANDARD_ALUMNI" -UsageLocation US

Error
Set-MsolUserLicense : A parameter cannot be found that matches parameter name 'UsageLocation'.

Really, Microsoft? Is it a required parameter or is it not a parameter? Make up your minds!

Turns out, it’s a parameter of the Set-MsolUser cmdlet:

PowerShell Commands
Set-MsolUser -UserPrincipalName "johndoe@example.com" -UsageLocation US
Set-MsolUserLicense -UserPrincipalName "johndoe@example.com" -AddLicenses "exampletenant:EXCHANGESTANDARD_ALUMNI"

The license was applied successfully.

Tags: , ,

Thursday, March 24th, 2016 Powershell No Comments

Querying an Oracle database from Powershell

I needed to query Oracle for information to use in a Powershell script. I found the following comprehensive blog post with details on how to do so using ODP.NET: Use Oracle ODP.NET and PowerShell to Simplify Data Access

The article is extremely long and goes into a lot of depth. The following is a short summary of the first steps, with just enough to get started:

  1. Download Oracle Data Provider for .NET (ODP.NET). (If that link doesn’t work just search for “Oracle ODP.NET”.)
    • Select “Download the latest ODP.NET production release.”
    • Select “64-bit ODAC Downloads”
    • Select “ODP.NET_Managed_ODAC12cR4.zip”
  2. Extract the ZIP file to C:\, which creates C:\ODP.NET_Managed_ODAC12cR4.
  3. Run cmd as administrator, navigate to C:\ODP.NET_Managed_ODAC12cR4, and run:
    install_odpm.bat C:\oracle\instantclient_10_2 both

In Powershell, add the DLL and set up a database connection and a query:

Add-Type -Path "C:\Users\chris\ODP.NET_Managed_ODAC12cR4\odp.net\managed\common\Oracle.ManagedDataAccess.dll"
$username = Read-Host -Prompt "Enter database username"
$password = Read-Host -Prompt "Enter database password"
$datasource = Read-Host -Prompt "Enter database TNS name"
$query = "SELECT first_name, last_name FROM users.user WHERE first_name = 'Chris' ORDER BY last_name"
$connectionString = 'User Id=' + $username + ';Password=' + $password + ';Data Source=' + $datasource
$connection = New-Object Oracle.ManagedDataAccess.Client.OracleConnection($connectionString)
$connection.open()
$command=$connection.CreateCommand()
$command.CommandText=$query
$reader=$command.ExecuteReader()
while ($reader.Read()) {
$reader.GetString(1) + ', ' + $reader.GetString(0)
}
$connection.Close()

Output of the above:

Brown, Chris
Carter, Chris
Jones, Chris
Smith, Chris

Tags: ,

Thursday, March 17th, 2016 Oracle, Powershell No Comments

Downgrading a Debian package

After I updated a Debian Wheezy server to Debian Jessie, I was having some problems with VLC. I was using the cvlc to capture and record video streams, but the capture would fail after 2 seconds. A colleague suggested that I leave the OS version alone, but downgrade VLC to confirm that the new version of VLC was causing the problem. A sound idea, but how do you downgrade a Debian package?

I found a variety of helpful sites and came up with the following:

  1. Add the source for the downgraded package the apt config
  2. Specify the target release in the apt config
  3. Use “pinning” to tell apt to use the older versions for the package and its dependencies
  4. Use apt-get install to install the dependencies and the package

Details as follows:

Add the source for the downgraded package the apt config
In this case, I needed to add the
I left my /etc/apt/sources.list as-is and added the following to /etc/apt/sources.list.d/vlc.list:

deb http://mirror.cc.columbia.edu/debian/ wheezy main non-free contrib
deb-src http://mirror.cc.columbia.edu/debian/ wheezy main non-free contrib

deb http://security.debian.org/ wheezy/updates main contrib non-free
deb-src http://security.debian.org/ wheezy/updates main contrib non-free

deb http://mirror.cc.columbia.edu/debian/ wheezy-updates main contrib non-free
deb-src http://mirror.cc.columbia.edu/debian/ wheezy-updates main contrib non-free

I used Columbia’s mirror because it’s fast and geographically nearer than most of the other mirrors. I’m sure not all of those sources were necessary (there are no security-related packages in VLC) but it didn’t hurt anything to include them.

Specify the target release in the apt config
I added the following to /etc/apt/apt.conf.d/80targetrelease to specify that I want Jessie to be the target/default release:

APT::Default-Release "jessie";

Use “pinning” to tell apt to use the older versions for package and its dependencies
I created the file /etc/apt/preferences.d/vlc containing the following. I added to the list of packages as apt-get install failed due to missing dependencies. The pin-priority 1001 was suggested by several sites to be used only in the case of downgrading a package.

Package: vlc
Pin: release n=wheezy
Pin-Priority: 1001

Package: vlc-data
Pin: release n=wheezy
Pin-Priority: 1001

Package: vlc-nox
Pin: release n=wheezy
Pin-Priority: 1001

Package: libvlc5
Pin: release n=wheezy
Pin-Priority: 1001

Package: libvlccore5
Pin: release n=wheezy
Pin-Priority: 1001

Package: vlc-plugin-notify
Pin: release n=wheezy
Pin-Priority: 1001

Package: vlc-plugin-pulse
Pin: release n=wheezy
Pin-Priority: 1001

Package: xdg-utils
Pin: release n=wheezy
Pin-Priority: 1001

Use apt-get install to install the dependencies and the package
I started by trying to apt-get install vlc but it complained about missing dependencies. I added the dependencies to the preferences file as describe in the section above, and then was able to install the following packages from Debian Wheezy:

apt-get install libvlccore5
apt-get install vlc-data
apt-get install libvlc5
apt-get install vlc-nox

Alternatively, I believe I could have skipped the preferences file and setting the pin-priorities by specifying the target Debian version when running apt-get install:

apt-get install -t wheezy libvlccore5
apt-get install -t wheezy vlc-data
apt-get install -t wheezy libvlc5
apt-get install -t wheezy vlc-nox

However, I’m not sure that the desired package version would be preserved after running apt-get dist-upgrade.

References
The following sites were helpful to me while I was figuring out how to do this, and if you are interested in pin-priority and the different values to use in different scenarios, I definitely recommend the first link:

Tags: , , ,

Thursday, February 25th, 2016 Uncategorized No Comments

Applying per directory X-Frame-Options headers in Apache

To help prevent against click-jacking, I had applied the following to my Apache 2.2 configuration based on the suggestions described in OWASP’s Clickjacking Defense Cheat Sheet and Mozilla Developer Network’s The X-Frame-Options response header:

Header always append X-Frame-Options SAMEORIGIN

However, my site has certain pages that are included in an iframe on another site, for the purpose of displaying content on digital signage devices. After I added that header, those pages would no longer load in an iframe on the digital signage devices’ browsers.

I thought I might be able to change SAMEORIGIN to ALLOW-FROM and list both the URI of my site and the URI of the digital signage page. However, the HTTP Header Field X-Frame-Options RFC indicates:

Wildcards or lists to declare multiple domains in one ALLOW-FROM statement are not permitted

The pages I wanted to exempt from the X-Frame-Options restriction exist in their own directory, /digitalsignage, so I tried to override the X-Frame-Options header in a .htaccess file:

Header always append X-Frame-Options ALLOW-ACCESS http://example.com

That caused a 500 Server Error. This message appeared in the error logs:

.htaccess: error: envclause should be in the form env=envar

The Header directive must be malformed, but I’m am not sure how. I did not determine how to properly format the statement so as not to produce that error, although several sites have pointed out that some browsers (Chrome, Safari) do not support ALLOW-ACCESS.

I changed the .htaccess file back to SAMEORIGIN, to match what was in the main site configuration:

Header always append X-Frame-Options SAMEORIGIN

I then noted that the response header sent by the server included SAMEORIGIN twice:

Header: SAMEORIGIN, SAMEORIGIN

That’s the expected behavior when using append. It appeared only once after I changed append to set:
Header always set X-Frame-Options SAMEORIGIN

I tried using set instead of append with ALLOW-ACCESS:

Header always set X-Frame-Options ALLOW-ACCESS http://example.com

But it still produced the same 500 Server Error.

After reading the documentation for Apache’s mod_headers, I realized that unset would allow me to remove the X-Frame-Options header from the /digitalsignage directory:
Header always unset X-Frame-Options

That worked, and the pages were successfully included as iframes in a page on the digital signage company’s site.

Tags: , , , , , ,

Wednesday, February 24th, 2016 Uncategorized 2 Comments

Canvas enrollments.csv and add_sis_stickiness

I have an enrollments.csv file for Instructure’s Canvas LMS, and I want all of the enrollments in it to “stick”–that is, to survive a batch mode SIS import. These are primarily course designers, and so they have no official standing in the class–and therefore are not in our database, and therefore are not included with regular updates to enrollments.

According to the Canvas documentation for SIS imports:

add_sis_stickiness – Boolean

This option, if present, will process all changes as if they were UI changes. This means that “stickiness” will be added to changed fields. This option is only processed if ‘override_sis_stickiness’ is also provided.

Source: https://canvas.instructure.com/doc/api/sis_imports.html#method.sis_imports_api.create

However, experience tells me otherwise. An inquiry to Instructure’s support confirms that add_sis_stickiness does not apply to enrollments. Enrollments added this way will be deleted following the next enrollments batch import.

The choices to preserve these course designer enrollments are basically to add each one manually using the web UI, or add them via the API. Either option will make the enrollments “stick.”

I opted to use the API. Since I already had a formatted input file, I wrote a short BASH script (with the help of several man pages and a couple StackOverflow pages) that reads the CSV and processes each row, adding the enrollment via the API:

headerrow=1
while read row; do
    if [ $headerrow -eq 0 ]
    then
        # get the SIS course ID
        cid="$(echo $row | cut -d',' -f1)"
        # get the SIS user ID
        uid="$(echo $row | cut -d',' -f2)"
        # get the role / enrollment type
        type="$(echo $row | cut -d',' -f3)"
        # reformat the enrollment type
        tid="$(echo $type | cut -c1 | tr [[:lower:]] [[:upper:]])""$(echo $type | cut -c2-)"Enrollment
        echo course is $cid
        echo user is $uid
        echo type is $tid
        result="$(curl https://[yourcanvassite].instructure.com/api/v1/courses/sis_course_id:$cid/enrollments -H 'Authorization: Bearer [REDACTED]' -X POST -F enrollment[type]=$tid -F enrollment[user_id]=sis_user_id:$uid -F 'enrollment[enrollment_state]=active' -F 'enrollment[notify]=false')"
        echo $result
    fi
    headerrow=0
done <enrollments.csv

Tags: , , ,

Friday, January 15th, 2016 Uncategorized No Comments

MySQL date_add and date_sub functions running against millions of rows

One of my servers runs a query once a week to remove all rows from a Syslog table (>20,000,000 rows) in a MySQL database that are older than 60 days. This was running terribly slowly and interfering with other tasks on the server.

Although the original query used a DELETE statement I’ve used SELECT statements in the examples below.

SELECT COUNT(*)
FROM SystemEvents
WHERE ReceivedAt < DATE_SUB(NOW(), INTERVAL 60 DAY);

That selects about 900,000 rows and takes about 45 seconds.

SELECT COUNT(*)
FROM SystemEvents
WHERE ReceivedAt < DATE_ADD(CURRENT_DATE, INTERVAL -60 DAY);

Likewise takes about 48 seconds.

Is MySQL running a function every time it makes a comparison? I decided to try using a hard-coded date to find out:

SELECT COUNT(*)
FROM SystemEvents
WHERE ReceivedAt < '2015-11-12 12:00:00';

6 seconds! Much faster.

I created a user-defined variable:
SET @sixty_days_ago = DATE_SUB(NOW(), INTERVAL 60 DAY);

Then ran the query:
SELECT COUNT(*)
FROM SystemEvents
WHERE ReceivedAt < @sixty_days_ago;

12 seconds. No 6 seconds, but still a fraction of the original time!

Tags: , ,

Monday, January 11th, 2016 MySQL, Tips & Tricks No Comments

Yammer Enterprise Block Users appears to be limited to 70 users per submission

This is just based on trial-and-error testing, but when I submit over 70 e-mail addresses at:

https://www.yammer.com/[my domain]/admin/blocked_email_addresses

I get one of the following errors:

  • Nginx 502 Bad Gateway
  • We’re sorry, but something went wrong.
    We’ve been notified about this issue and we’ll take a look at it shortly.

When I submit 70 addresses or fewer, it works as expected.

I have not yet discovered a way to block addresses via PowerShell.

Tags: ,

Friday, December 18th, 2015 Uncategorized 1 Comment

Error retrieving Glacier vault inventory via AWS CLI: Unknown options: inventory-retrieval}’

Amazon’s Glacier service is great, but low-cost storage has other costs. For example, you might have a vault but not know what archives it contains. You can retrieve the vault inventory–essentially listing the contents of a directory–but like any Glacier retrieval, it may take several hours.

Using Downloading Vault Inventory using the REST API, I put this together:
C:\>aws glacier initiate-job --account-id - --vault-name my_poorly_named_vault --job-parameters '{"Type": "inventory-retrieval"}'

Which returned:

Unknown options: inventory-retrieval}'

I had taken the command directly from AWS’s example:
http://docs.aws.amazon.com/cli/latest/reference/glacier/initiate-job.html.

(Keep in mind that I had already followed the steps at Installing the Amazon Web Services Command Line Interface and Configuring the Amazon Web Services Command Line Interface.)

According to the documentation for the job-parameters option (http://docs.aws.amazon.com/sdkforruby/api/Aws/Glacier/Types/JobParameters.html#type-instance_method),
valid values are “archive-retrieval” and “inventory-retrieval”.

But the error message says inventory-retrieval}’. Why is it picking up the trailing curly brace and the apostrophe?

I formatted the job-parameters JSON in a file named aws-json.txt, with the curly braces on separate lines:

{
    "Type": "inventory-retrieval"
}

I tried this variation on the initiate-job command:
C:\>aws glacier initiate-job --account-id - --vault-name my_poorly_named_vault --job-parameters file://aws-json.txt

That worked!

The results returned:

{
    "jobId": "y8ugyoNzzusaf6Lv72G3hsjAA6O7nw5bJQ2u6J9TDnJ82_qx-lxnqrhSxIcGvOU1iiXoUhZboiojxsDu8gLQOfiJ7hR2",
    "location": "/123456789011/vaults/my_poorly_named_vault/jobs/y8ugyoNzzusaf6Lv72G3hsjAA6O7nw5bJQ2u6J9TDnJ82_qx-lxnqrhSxIcGvOU1iiXoUhZboiojxsDu8gLQOfiJ7hR2"
}

Tags: , , ,

Wednesday, December 2nd, 2015 Uncategorized No Comments