Size of data in bytes

This was prompted by an error I was running into with the AWS s3 service: I needed to tell the transfer utility the size of the data, in bytes, when transferring large files.

In this case I am looking at files of characters. Some of these methods should work equally well for binary files, and others don’t. In the following examples, I’ll use the full text of Moby-Dick from Project Gutenberg, 2701-0.txt, as the target file. I retrieved the file using the following command:

curl -O http://www.gutenberg.org/files/2701/2701-0.txt

A couple commands to get size in bytes immediately came to mind: ls, stat, and wc.

$ ls -l 2701-0.txt | cut -d' ' -f5
1276201

$ stat --format %s 2701-0.txt 
1276201

$ wc -c 2701-0.txt | cut -d' ' -f1
1276201

All those options work. But what if the input isn’t a file on disk, and instead is an input stream? This is to demonstrate counting the bytes in a character stream coming from any source, so forgive the “useless use of cat”:

$ cat 2701-0.txt | wc -c
1276201

$ cat 2701-0.txt | cksum | cut -d' ' -f2
1276201

$ cat 2701-0.txt | dd of=/dev/null
2492+1 records in
2492+1 records out
1276201 bytes (1.3 MB, 1.2 MiB) copied, 0.00997434 s, 128 MB/s

The output from dd above is not the simplest thing to parse. It’s multi-line and sent to stderr, so I redirected it to stdout and grepped for “bytes”:

$ cat 2701-0.txt | dd of=/dev/null 2>&1 | grep 'bytes' | cut -d' ' -f1
1276201

There are at least 5 methods to find the size of a file using common command-line tools:

  • ls
  • stat
  • wc
  • cksum
  • dd

Know of others? Leave a comment below.

nmap scans the top 1000 ports by default, but which 1000?

From man nmap:

The simple command nmap target scans 1,000 TCP ports on the host target.

You might reasonable ask, which 1,000 ports is it? Is the particular port in which I am interested included?

Fortunately, nmap has a list of ports/services that includes how frequently they are used. From this we can get the top 1000:

grep -v '^#' /usr/share/nmap/nmap-services | sort -rk3 | head -n1000
  • The initial grep is to filter out the comments (lines that begin with the hash mark).
  • The sort command sorts in descending order, by the 3rd column (the frequency).
  • The final head command displays only the top 1000 results.

In my cases, I wondered if the radmin port, 4899/tcp, was included in an nmap scan. I piped the above command to grep to find out:

grep -v '^#' /usr/share/nmap/nmap-services | sort -rk3 | head -n1000 | grep 4889
radmin  4899/tcp        0.003337        # Radmin (www.radmin.com) remote PC control software

It is included in a default nmap scan.

Is there an easier way to do this? Drop me a line in the comments!

Running VMs? Delete wireless packages!

A best practice for system configuration is to remove any unneeded software. It’s sometimes difficult to know exactly what is needed and what isn’t, but CentOS 7 minimal and CentOS 8 minimal both install a number of packages related to wireless networking. If you’re running a server or a VM there’s almost never a need for these to be present.

To identify packages, I used yum search (substitute dnf for yum on CentOS 8):

yum search wireless

I used the same command a redirected the output to a file:

yum search wireless >wireless_packages

To get just the package names and convert it to a space-separated list, I used grep, cut, and paste:

grep -v Summary wireless_packages | cut -d. -f1 | paste -d' ' -s

You can remove them with the following command:

sudo yum remove iw iwl6000-firmware crda iwl100-firmware iwl1000-firmware iwl3945-firmware iwl4965-firmware iwl5000-firmware iwl5150-firmware iwl105-firmware iwl135-firmware iwl3160-firmware iwl6000g2a-firmware iwl6000g2b-firmware iwl6050-firmware iwl2000-firmware iwl2030-firmware iwl7260-firmware

iw and crda were not installed, so were ignored. The rest were removed.

This may seem trivial, but it frees up some disk space (~100MB) and it means that these packages won’t need to be updated in the future. Getting notifications from your monitoring systems or vulnerability management systems about updates or security updates to unused and unnecessary packages should be avoided.

Combining pcap (packet capture) files

Motivation: I wanted to combine 2 or more packet capture, or pcap, files in order to create an example:

  • One that contains just malicious (or simulated malicious) network traffic
  • Another contains legitimate, non-malicious network traffic

Many example packet capture files focus either specifically on malware, exploits, C2 traffic, etc. (like Security Onion’s list of PCAPs for Testing) or on examples of legitimate traffic (like Wireshark’s Sample Captures). I wanted to create an example that would interweave such sources and intersperse malicious and legitimate traffic, as they would typically occur concurrently.

In addition to tcpdump, there are three CLI tools provided by Wireshark that I used to help accomplish this:

  • capinfos – provides high-level data about a packet capture file
  • mergecap – combines 2 or more packet capture files
  • editcap – modified packet details, such as timestamps, in a packet capture file

Continue reading Combining pcap (packet capture) files

Using Buildah to build containers for Docker and Podman

One of my colleagues pointed me to an article on using Buildah to create container images: How rootless Buildah works: Building containers in unprivileged environments.

I decided to test it out! In this case, I just wanted to build a container using the shell script method described in the article, rather than using a Dockerfile. Although the rootless aspect is interesting to me, I believe that requires newer versions of Buildah than what is available by default on CentOS 7. A project for another day!

First, I needed a test case. I decided to use my NLTK chatbot container image, which can be found at:

Continue reading Using Buildah to build containers for Docker and Podman

Modifying a packet capture with Scapy

My motivation was to start from a known good packet capture, for example, a DNS request and reply, and modify that request to create something interesting: an example to examine in Wireshark, or positive and negative test cases for an IDS software (Snort, Suricata).

I haven’t done much with Scapy before, but it seemed like the right tool for the task. My planned steps were as follows:

  1. Take pcap (packet capture)
  2. Import pcap via scapy
  3. Modify pcap
  4. Export pcap
  5. View pcap in Wireshark

Continue reading Modifying a packet capture with Scapy

Notifying a REST API from Icinga2

I wanted to send Icinga2 notifications to Slack. Some hosts and services don’t rise to the level of a PagerDuty notification, but e-mail just doesn’t cut it. A message in a Slack channel seemed an appropriate in-between.

This process is relatively straightforward, although I ran into some issues with SELinux that I will cover in this post.
Continue reading Notifying a REST API from Icinga2

Running WordPress on Docker

Similar to the previous post, Running Joomla on Docker, I was interested in spinning up a temporary WordPress installation so that I could target it with various scanning and reconnaissance tools. There is an official WordPress Docker image at https://hub.docker.com/_/wordpress/.

The steps were more-or-less the same. Note that if you followed the steps in the previous post, you will likely want to stop and remove the existing MySQL container before attempting to start a new one with the same name:

docker stop some-mysql
docker rm some-mysql

Start the MySQL Docker container:

docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=passW0rd -e MYSQL_DATABASE=wordpress -e MYSQL_USER=wordpress -e MYSQL_PASSWORD=wpP455 -d mysql:5

Start the WordPress Docker container:

docker run --name some-wordpress --link some-mysql:mysql -e WORDPRESS_DB_HOST=172.17.0.2 -e WORDPRESS_DB_USER=wordpress -e WORDPRESS_DB_PASSWORD=wpP455 -e WORDPRESS_DB_NAME=wordpress -p 8080:80 -d wordpress

I was then able to visit http://localhost:8080 and complete the web-based setup tasks.

Note that the MySQL container, as launched, does not have any shared volumes. Everything stored there is ephemeral and will be lost if the container is removed. To my surprise, however, the content survived stopping and restarting the container. The volumes for each container are located in the following directory:

/var/lib/docker/volumes/

Using docker inspect some-wordpress I could see that there was a mounted volume at:

/var/lib/docker/volumes/be3d54591da609e911a1ec3f0615a564990b37da184a67fab0ac0e75cc711c7f/_data

Indeed, the usual WordPress files, such as wp-config.php, were located there.

I did the same for the MySQL container and found the .frm and .ibd files for each of the tables in the WordPress database.

These files persist when the container is stopped, and persist even when the container is removed! In fact, when I removed all containers, I discovered there were still 22 volumes in /var/lib/docker/volumes from previous container projects and experiments.

The command to view these volumes is:

docker volume ls

To remove unused volumes, use:

docker volume prune

Container volumes are not as ephemeral as I originally thought!

Running Joomla on Docker

I was looking for a well-known CMS (Content Management System) that I could easily run in a Docker container as a target for information security reconnaissance tools, such as WhatWeb.

I found an official Docker image for Joomla, a CMS that I had used previously some years ago: https://hub.docker.com/_/joomla
Continue reading Running Joomla on Docker

Python Flask, escaping HTML strings, and the Markup class

As in the previous post, I had created a simple web app using Python Flask to use as a teaching tool. The purpose was to demonstrate SQL injection and XSS (cross-site scripting) vulnerabilities and how to remediate them.

In this case, the remediation step for XSS (escaping output) tripped me up. I tried this:

return '<p>You searched for: ' + escape(user_input) + '</p>'

I expected it to escape only the user_input variable, but instead it escaped all the HTML, returning this:

&lt;p&gt;You searched for: &lt;script&gt;alert(1)&lt;/script&gt;&lt;/p&gt;

Continue reading Python Flask, escaping HTML strings, and the Markup class