Monthly Archives: August, 2014

Python Adventures – 05

Following the free E-book Learning Python the Hard Way and getting down the basics of Python. In Exercise 1: A Good First Program
it’s how to write your very first Hello World! program.

 

Bring up Canopy in Ubuntu or Windows and start a new script.

 

# -*- coding: utf-8 -*-

print “Hello there world!”

print “Hello back to you!”

print “I like typing with you.”

print “Typing back is fun.”

print ‘Yay! Printing lines with Python.’

print ‘Um, I’m not touching “that!”‘

print ‘I thought you “liked” Pythons? “wink, wink, nudge nudge” ‘

print ‘The language Python and that dear Sir is definitely not a “Python”!’

 

If all goes well you should see a similar result:


 

My PowerShell acted a little funny and changed some characters:


 

However, with the assistance of a colleague with ServerCare.nl, a PowerShell Guru. I learned an important lesson. Characters might look the same but are not. For instance the < ” > I used versus what is on a website. I tried at first to type out the script but it didn’t work. I kept getting syntax errors. I copied from the exercise website, it worked in Canopy. I rewrote everything except the quote structure, it worked in Canopy. When I ran it in PowerShell, strange characters. As in real life, one sometimes must employ escape techniques around a Python. These are called escape characters so a “double quote is just a” .

 

The corrected Python:

# -*- coding: utf-8 -*-

print “Hello there world!”

print “Hello back to you!”

print “I like typing with you.”

print “Typing back is fun.”

print ‘Yay! Printing lines with Python.’

print “Um, I\’m not touching that!”

print “I thought you \”liked\” Pythons? \”wink, wink, nudge nudge\” ”

print ‘The language Python and that dear Sir is definitely not a \”Python\”!’

 

The corrected PowerShell output:


 

I had to jump ever so slightly beyond this exercise to Exercise 10: What Was That? To get a better handle on escape characters.

 

Off to the next exercise and adding in fun with data.

 

 

 

 

 

 

Python Adventures – 04

Getting a data set to play with!!!!!

The Data-Driven Security book uses free tools and sources. In Chapter 3 the fun begins by gathering some data. The authors begin with AlienVault’s open source reputation database. It’s updated hourly and I noticed sometimes not available on the hour possibly due to traffic load.

I used the following Python script, stored also on GitHub (to be linked) to get it the AlienVault data:

# -*- coding: utf-8 -*-

# Similar to Listing 3-1

import os

import sys

os.chdir(os.path.expanduser(“~”) + “/book/ch03”)

 

# Similar to Listing 3-3

# URL for the AlienVault IP Reputation Database (OSSIM format)

# storing the URL in a variable makes it easier to modify later

# if it changes. NOTE: I choose a direct URL because I don’t

# know any better J.

 

import urllib

import os.path

 

avURL = “http://reputation.alienvault.com/reputation.data&#8221;

 

# relative path for the downloaded data

avRep = “D:\Python\Data Driven Security/reputation.data”

 

To update once a day, I scheduled a Windows task. This is only the start! My version of Windows can’t update more often via the GUI and I should be able to improve this so I can script what I need.

I scheduled the task to run a script daily at 39 minutes after a random hour.

# Similar to Listing 3-3

# URL for the AlienVault IP Reputation Database (OSSIM format)

# storing the URL in a variable makes it easier to modify later

# if it changes. NOTE: I choose a direct URL because I don’t

# know any better J.

 

import urllib

import os.path

avURL = “http://reputation.alienvault.com/reputation.data&#8221;

 

# relative path for the downloaded data

avRep = “D:\Python\Data Driven Security/reputation.data”

#I’m sure there are much better ways of accomplishing this, comments warmly welcomed!

 

I am puzzled about the lack of HTTPS and/or a more secure method available to feed in trusted, verified reputation data both directly to a SIEM like OSSIM or via the website. I used a tool called EvilGrade way too much in the past (with permission) which makes me twitchy regarding insecure updates. An awesome write up on pivoting attacks with EvilGrade.

 

I’m certain there are better methods of scripting this. I made a basic variation from the DDS book. I want to improve on this script and comments and ideas are welcomed.

 

 

 

 

 

 

Python Adventures – 03

Sharing code and delving into Python & Pandas.

My GitHub repository (which I must reorganize prior to sharing a link) where I will endeavor to post a copy of all the code I create for my Python Adventures. I look forward to open reviews and improvements.

 

The Python & Pandas example based on Data-Driven Security Listing 2-2:

#Python & Pandas data frame example similar to Data-Driven Security Listing 2-2

# create a new data frame

import numpy as np

import pandas as pd

 

#create a new data frame of 5 IT and OT assets and vulnerability counts

assets_df = pd.DataFrame( {

“name” : [ “ControlRoom-PC001″,”PLC-002″,”RTU-003″,”DCS-004″,”FilePrint-SVR005” ],

“os” : [ “WinXP”,”Fatek”,”GE_D20MX”,”DLink_DCS-2000″,”W2K8″ ],

“highvulns” : [ 25,5,12,6,0 ]

} )

 

#review the data frame structure & content

print(assets_df.dtypes)

assets_df.head()

 

#shows a sample or slice of the available operating systems input

assets_df.os.head()

 

#Addition of a new column with IP address information & new column

assets_df[‘ip’] = [ “10.10.1.2”,”10.10.2.2″,”10.10.3.3″,

“10.10.2.4”, “10.10.4.5”]

 

#Display assets only with greater than 10 high vulnerabilities & new column

assets_df[assets_df.highvulns>10].head()

 

#Categorize assets in zones and add a new column

assets_df[‘zones’] = np.where(

assets_df.ip.str.startswith(“10.10.2”), “Zone1”, “Zone2”)

 

#final inspection of code input

assets_df.head()

 

If all goes well the output should look something like:

What is nice about Python with Pandas is the nice output layout. When I do the same thing in R, no table layout without work, more difficult to read through. This would be of greater importance when handling larger sets of data and limited knowledge of Python and R.

Why did I pick Operations Technologies (OT) and IT resources? OT is what runs much of our physical world nowadays, not shiny IT like new servers or laptops. IT controls our virtual selves, our data and not for example traffic lights, railway signaling, medical equipment or factories. For this reason, I will be presenting Confessions of an IT/OT Hacker at the European Industrial Control Systems Summit, London, UK Royal Aeronautical Society, 22 & 23 September, 2014. Everyone in IT and should familiarize themselves with the basics of Industrial Control Systems.

 

Python Adventures – 02

The noob in me means I should read the instructions first, the engineer in me says I can figure it out, I don’t need no stinking instructions! How quickly I forget the last time I attempted this method with Ikea kitchen cabinets, um…..Moving swiftly along; I fixed my Windows RStudio installation issues. I had this strange assumption that RStudio would come with R. Similarly to how Visual Studio comes with C#. Assumptions and IT rarely work out well.

R goes hand in hand with Python if you want to break out of metrics beyond averages, using a normal distribution or standard deviation. If you want to crunch juicy, more advanced numbers R is the way to go. I’m new to R and I know just enough statistics to be slightly mathematically dangerous J

Remember, numbers are your friend, they justify the return on IT Security investment, i.e. your paycheck.

To download R, go to the CRAN project page and choose a close mirror for the newest package which is R-3.1.1 for Windows 32/64. Although the title of the package screams security vulnerabilities, my version was patched to 2014-08-18, the day I downloaded it. Once R is downloaded and installed, RStudio can be installed and it works straight away on Windows.

Let’s say I have 5 assets and I want to put them in a data frame with vulnerability counts:

#R data frame example similar to Data-Driven Security Listing 2-1

#create a new data frame of 5 IT and OT assets and vulnerability counts

assets.df <- data.frame(

name=c(“ControlRoom-PC001″,”PLC-002″,”RTU-003″,”DCS-004″,”FilePrint-SVR005”),

os=c(“WinXP”,”Fatek”,”GE_D20MX”,”DLink_DCS-2000″,”W2K8″),

highvulns=c(25,5,12,6,0))

#review the data frame structure & content

str(assets.df)

#review assets as now added in

head(assets.df)

#shows a sample or slice of the available operating systems input

head(assets.df$os)

#Addition of a new column with IP address information & new column

assets.df$ip <- c(“10.10.1.2″,”10.10.2.2″,”10.10.3.3”,

“10.10.2.4”, “10.10.4.5”)

#Display assets only with greater than 10 high vulnerabilities & new column

head(assets.df[assets.df$highvulns>10,])

#Categorize assets in zones and add a new column

assets.df$zones <- ifelse(grepl(“^10.10.2″,assets.df$ip),”Zone1″,”Zone2”)

#final inspection of code input

head(assets.df)

 

If all goes well your run output will look like this:

>

 #R data frame example similar to Data-Driven Security Listing 2-1
> #create a new data frame of 5 IT and OT assets and vulnerability counts
> assets.df <- data.frame(
+   name=c("ControlRoom-PC001","PLC-002","RTU-003","DCS-004","FilePrint-SVR005"),
+   os=c("WinXP","Fatek","GE_D20MX","DLink_DCS-2000","W2K8"),
+   highvulns=c(25,5,12,6,0))
> 
> #review the data frame structure & content
> str(assets.df)
'data.frame':    5 obs. of  3 variables:
 $ name     : Factor w/ 5 levels "ControlRoom-PC001",..: 1 4 5 2 3
 $ os       : Factor w/ 5 levels "DLink_DCS-2000",..: 5 2 3 1 4
 $ highvulns: num  25 5 12 6 0
> #review assets as now added in
> head(assets.df)
               name             os highvulns
1 ControlRoom-PC001          WinXP        25
2           PLC-002          Fatek         5
3           RTU-003       GE_D20MX        12
4           DCS-004 DLink_DCS-2000         6
5  FilePrint-SVR005           W2K8         0
> #shows a sample or slice of the available operating systems input
> head(assets.df$os)
[1] WinXP          Fatek          GE_D20MX       DLink_DCS-2000 W2K8          
Levels: DLink_DCS-2000 Fatek GE_D20MX W2K8 WinXP
> #Addition of a new column with IP address information & new column
> assets.df$ip <- c("10.10.1.2","10.10.2.2","10.10.3.3",
+                   "10.10.2.4", "10.10.4.5") 
> #Display assets only with greater than 10 high vulnerabilities & new column
> head(assets.df[assets.df$highvulns>10,])
               name       os highvulns        ip
1 ControlRoom-PC001    WinXP        25 10.10.1.2
3           RTU-003 GE_D20MX        12 10.10.3.3
> #Categorize assets in zones and add a new column
> assets.df$zones <- ifelse(grepl("^10.10.2",assets.df$ip),"Zone1","Zone2")
> #final inspection of code input
> head(assets.df)
               name             os highvulns        ip zones
1 ControlRoom-PC001          WinXP        25 10.10.1.2 Zone2
2           PLC-002          Fatek         5 10.10.2.2 Zone1
3           RTU-003       GE_D20MX        12 10.10.3.3 Zone2
4           DCS-004 DLink_DCS-2000         6 10.10.2.4 Zone1
5  FilePrint-SVR005           W2K8         0 10.10.4.5 Zone2
>

 

 

 

 

 

 

 

 

 

 

 

Python Adventures – 01

I completed Learn Python the Hard Way Exercise 0: The Setup & Appendix A: Command Line Crash Course by Zed A. Shaw. The command line section was a refresher but I’m unfamiliar with using PowerShell vs a command prompt in Windows and I never used pushd and popd before. Something new is always cool. There was one caveat with my PowerShell: -p didn’t work for me, I had to use –path instead.

I figured it best to download the Data-Driven Security book code from Wiley as I’m prone to typos. That way I can test the clean, working code if my results fail epically. Prior to moving forward to chapter 3 of the book, one must delve deep into the Data Frame. The book code had Python Listing 2-2 but I’m having trouble with 2-1, 2-3 & 2-4 due to my Windows RStudio installation.

Nothing helps installation frustration like reference materials J

Short Introductions to Python, Pandas and R references:

 

Learn Python in 10 minutes

 

 


by Stavros Korokithakis

10 Minutes to Pandas by the Pandas Development Team

A (Very) Short Introduction to R by Paul Torfs & Claudia Brauer SHA256: d847c553386deaf8e85a718c91ef5ec122d31d3faf4c291b5a1f6e1ceb8ab5d2

The R Markdown cheat sheet by RStudio

 

 

 

Python Adventures – Setup

I’m following the book Learn Python the Hard Way, recommended by @stevemcgrath. I want to tackle some serious data for security analytics using Python and R as well. Ultimately, I wish to create some cool, easy to understand visualizations. The main goal is to complete the book Data-Driven Security and kick some serious security data analytics.

First, I started by installing Canopy 64 bit on Windows 8.1 and Ubuntu 14.04. This sounds easy, it wasn’t. Neither OS version installation worked out of the box. I adjusted the graphics options in the Canopy main area, both OS versions via: Main Screen, Edit, Preferences, Python, Inline (SVG). I will show both operating systems were feasible.

I then ran the following verification check per Data-Driven Security:

 

import pandas as pd

import numpy as np

np.random.seed(1492)

test_df = pd.DataFrame({ “var1”: np.random.randn(5000) })

test_df.hist()


In Windows, I kept getting an openpyxl versioning error. This took a while to solve. After a few uninstall, re-install, “Kernel died, restarting” errors it all worked!

In Linux, I ran into matplotlib, openpyxl and fttype verison errors.

To solve fttype & matplotlib, I found a solution posted by user3888817 on Stack Exchange:

enpkg –no-deps matplotlib 1.2.1

enpkg –no-deps libpng 1.2.40

enpkg –no-deps freetype 2.4.4

 

To solve the openpyxl errors, I can’t remember where I found it:

sudo apt-get install mercurial

 

To install R, I went to R Studio Desktop Download for Windows

 

To install in Ubuntu I went to the Ubuntu Software Center, RStudio

To install ggplot2, at a terminal session:

 

sudo apt-get install r-base-core

R

In R:

install.packages(“ggplot2”)

 

To verify your R installation, run inside R:

 

library(ggplot2)

set.seed(1492)

test.df = data.frame(var1=rnorm(5000))

ggplot(data=test.df) + geom_histogram(aes(x=var1))

 


 

 

Python and R are now both installed!!! J

 

 

 

 

 

TSA, Opt-out and you’re a “Criminal Hacker” Yippee!

The continuing adventures of the Freedom Fondle and the nerve of some who choose to opt-out

Traveling to and from the USA, even for US citizens is a challenge. I’m getting used to the “random” SSSS on my boarding pass, intrusive and wholly inappropriate questions about my work, employer, ethnicity and religion. The accusations of carrying a fake passport because I have an “accent” or otherwise known as traveling whilst Hispanic in the USA. I travel with limited clothing as I expect them to be ripped or otherwise destroyed in-front of my eyes, again by Customs and Boarder Patrol. I stopped carrying anything which could even remotely be confused with the Arabic language. I travel with very limited, encrypted data. My family expects detainment and knows to contact a USA attorney if I don’t check in quickly enough after landing. Today was a new one and rather unexpected. I forgot to expect the unexpected with the TSA.

I opt-out when I’m traveling within the United States. This isn’t an option when flying from Europe to the USA due to an underwear obsessed, idiotic terrorist; but it is and a right whilst traveling within the USA boarders. As per usual I arrived in plenty of time for my flight, checked-in and got in the security theater TSA line for the shredding of my 4th amendment rights. As I approached the full body scanner I politely informed the male officer I wished to opt-out. Without engaging with any other ancillary officers, I waited patently to be freedom fondled in full public view. Standing up for your rights sometimes involves strangers groping my private parts, and I can live with that.

The female TSA officer by the scanner decided to loudly voice her option of those who opt-out. Standing by the full body, 4th amendment dissolving scanner. She explained to her male co-worker at a volume all in the area could clearly hear. A rant on how “all these criminals, so-called hackers, are a bunch of useless posers who should be in a jail cell not flying or pulling their BS by opting out”. For a few minutes she continued to spew her utter ignorance in an attempt to intimidate and humiliate me. I had no choice but to listen, the other passengers being screened had to as well. I wore no identifiable “hacker” shirt, just glasses and my usual pile of technology. My jacket was from an off-Broadway play, Avenue Q and I wore glasses. I guess glasses, computers and opting out is now a sure sign you are a criminal hacker that should be thrown into jail. I must have looked dangerous in my -7.00 bottle thick glasses!

Hopefully the situation will have a somewhat happy ending. When my freedom fondling by a different TSA officer began. I explained I wished to file a complaint, in writing, as soon as her glove was off. I was sent to a very understanding and sympathetic supervisor. After explaining I had absolutely no verbal or other engagement with the verbally abusive officer. I was given a form, the officer’s full name and a very friendly verbal acknowledgment that no TSA officer should act in such a verbally abusive manner. As many of the passengers on my flight heard the comments whilst being screened. I didn’t have to engage in any flight chit-chat. An added bonus for being labeled a criminal prior to boarding an airplane (?)

Not all the TSA are bad, just enough to taint the organization and cause disrepute to the actual honest hard-working agents. Hopefully my written complaint will be taken. Hackers are not criminals, nor are those who opt out. Those in government positions which chose to openly attempt to intimidate people into giving up their rights are.

We are the Calvary!