And play the mother’s part, kiss me, be kind;
So will I pray that thou mayst prove me.
Weary with toil, I haste me thence?
Till I return, of posting is no remedy,
It is the time with thoughts of love as oft as thou shalt find
Those children nursed, deliver’d from thy heart,
And take thou my love shall in these black lines be seen,
And they shall live, and he stole that word
From thy behaviour; beauty doth he give,
And found such fair assistance in my will no fair acceptance shine?
The sea, all water, yet receives rain still,
And then thou hast her it is built anew,
Grows fairer than at first, more strong, greater.


Number of key tuples: 14916
Mean Choices: 1.174175
Min Choices: 1
Max Choices: 27
Sum of Squares: 13581.492357
Standard Deviation: 116.539660


I have had a vague “todo” in my head for a while to do something with Markov text generation. One of the things that I look forward to when I scan through my spam folders is finding interesting text that was obviously generated. I was lazily browsing through some of my news feeds and following links when I came across this post titled Markov and You by Jeff Atwood. I especially liked the Garkov reference he used to illustrate the usage. I decided I would try to do something similar and found a basic markov implementation in python at this Usware blog post.

After playing with the ”quick brown fox” implementation and pulling in some CNN articles I decided to do something a bit more interesting and downloaded Shakespeare’s Sonnets from Project Gutenberg.

I modified my generator implementation to have output based on the number of lines emitted, and added a reseed function to select a new start tuple in the event that no keys matched my existing search tuple. Finally, I added some simple output statistics to get a feel for how the constructed database looked when I tweaked the chain length parameter. Having shorter chain lengths tended to make the text too random, while longer ones pulled in too much of an existing sonnet sequence. Watching the number of key tuples and the summary statistics of the choices helped me tune the code for this corpus.

The code is available here.

I saw a reference to ai-junkie somewhere and read through the “Agents” section and thought I would have a go at building a simple finite state machine game in Prolog based on the Miner Bob example found here, specifically in Figure 2.2.

I’m not sure if this is the best way to do this in Prolog, this approach will eventually fail for long enough games since the stack just keeps growing.

/*
  Simple FSM based on

http://www.ai-junkie.com/architecture/state_driven/tut_state1.html

  Player data structure looks like:
  player(wealth, thirst, banked).
*/

/* Game parameters */
goal(20).
maxThirst(3).
maxInventory(4).
rateThirst(1).
rateMine(2).

% debug helper which just dumps player information
debug(P) :-
  write(P),nl.

% The home state, where we start and end the game.
state(home, player(Wealth,Thirst,Banked)) :-
  write('Home'),nl,
  (Banked =:= 0 ->
    fire(state(home), event(rested), player(Wealth,Thirst,Banked));
    write('Retirement!'),nl,
    debug(player(Wealth,Thirst,Banked))).

% The bank state, where wealth is deposited and our banked amount checked
state(bank, player(Wealth,Thirst,Banked)) :-
  write('Bank'),nl,
  B is Wealth + Banked,
  P = player(0,Thirst,B),
  goal(G),
  (B < G ->
    fire(state(bank), event(not_wealthy), P);
    fire(state(bank), event(wealth), P)).

% The mining state where we do our work, generating wealth and thirst
state(mine, player(Wealth,Thirst,Banked)) :-
  write('Mining'),nl,
  rateMine(Rm),
  rateThirst(Rt),
  W is Wealth+Rm,
  T is Thirst+Rt,
  P = player(W,T,Banked),
  maxThirst(MaxThirst),
  maxInventory(MaxInv),
  (T >= MaxThirst -> fire(state(mine), event(thirsty), P);
    (W >= MaxInv ->
      fire(state(mine), event(deposit), P);
      fire(state(mine), event(not_wealthy), P))).

% The quenching state, wherein we slake our thirst.
state(quench, player(Wealth,_,Banked)) :-
  write('Drinking'),nl,
  fire(state(quench), event(not_thirsty), player(Wealth,0,Banked)).

% Definitions of our state transitions and event
transition(state(home),   event(rested),      state(mine)).
transition(state(mine),   event(deposit),     state(bank)).
transition(state(mine),   event(thirsty),     state(quench)).
transition(state(mine),   event(not_wealthy), state(mine)).
transition(state(quench), event(not_thirsty), state(mine)).
transition(state(bank),   event(wealth),      state(home)).
transition(state(bank),   event(not_wealthy), state(mine)).

% Fires a state transition event and transfers us to the next state.
fire(state(S), event(E), P) :-
  write('\tStatus: '), debug(P),
  transition(state(S),event(E),state(NextState)),
  state(NextState, P).

% Starts the game!
start :-
  P = player(0, 0, 0),
  state(home, P).

Running it looks like:

| ?- start.
Home
Status: player(0,0,0)
Mining
Status: player(2,1,0)
Mining
Status: player(4,2,0)
Bank
Status: player(0,2,4)
Mining
Status: player(2,3,4)
Drinking
Status: player(2,0,4)
Mining
Status: player(4,1,4)
Bank
Status: player(0,1,8)
Mining
Status: player(2,2,8)
Mining
Status: player(4,3,8)
Drinking
Status: player(4,0,8)
Mining
Status: player(6,1,8)
Bank
Status: player(0,1,14)
Mining
Status: player(2,2,14)
Mining
Status: player(4,3,14)
Drinking
Status: player(4,0,14)
Mining
Status: player(6,1,14)
Bank
Status: player(0,1,20)
Home
Retirement!
player(0,1,20)


This is a directed graph of a relational database schema. The node size reflects the count of rows within the table, and the node color reflects the out-degree measurement of the node, that is the number of outgoing references the node has. The node color is ramped from blue to red. I’ve left off node labels for confidentiality.

The most central highly connected node has a large number of incoming edges because it is used for system wide auditing purposes. Certain subsystems are able to be identified, and the ring of stand-alone nodes mainly represent setup and configuration tables that do not participate in cross-domain operations.

I investigated the filtering capabilities of Gephi and found them pretty easy to use, I was able to perform some basic k-core analysis of the node degrees to be able to identify the most linked tables in the schema. I think making an animation of this could prove to be informative.

One feature that I would like but have not determined whether it is possible with Gephi is to identify dominators and cliques.

I used Groovy + gexf4j to generate the GEXF XML graph. The script is specific for Oracle, but should be able to be easily modified to interrogate the metadata of other relational database systems.

/*
 Retrieve database metadata about table names and relationships.
 This information will be used to generate GEXL for use in Gephi.
*/

import groovy.sql.Sql
import com.ojn.gexf4j.core.data.*
import com.ojn.gexf4j.core.impl.*
import com.ojn.gexf4j.core.impl.data.*

def sql = Sql.newInstance(
	"jdbc:oracle:thin:@server:port:sid",
	"username",
	"password",
	"oracle.jdbc.OracleDriver"
)

def query_tables_and_count(sql) {
	def tables = []
	sql.eachRow("select * from user_objects where object_type = 'TABLE'") {
		def name  = it.OBJECT_NAME
		def countQuery = "select count(*) as cnt from ${name}".toString()
		def count = sql.firstRow(countQuery).cnt
		tables << ['name': name, 'count' : count]
	}
	tables
}

/**
 	@see http://www.conandalton.net/2008/09/list-foreign-key-constraints-in-oracle.html
*/
def query_foreign_relationships(sql) {
	String foreignKeyQuery = """
	select
		distinct col.table_name as key,
		rel.table_name as references
	from
		user_tab_columns col
		join user_cons_columns con
		  on col.table_name = con.table_name
		 and col.column_name = con.column_name
		join user_constraints cc
		  on con.constraint_name = cc.constraint_name
		join user_cons_columns rel
		  on cc.r_constraint_name = rel.constraint_name
		 and con.position = rel.position
	where
		cc.constraint_type = 'R'
	order by
		col.table_name, rel.table_name
	"""
	def results = []
	sql.eachRow(foreignKeyQuery) {
		results << ['key':it.key, 'ref':it.references]
	}
	results
}

def write_xml(sql) {
	def gexf = new GexfImpl()

	// Metadata
	gexf.getMetadata()
		.setLastModified(Calendar.getInstance().getTime())
		.setCreator("Jason B. Smith")
		.setDescription("DB Graph")

	attrList = new AttributeListImpl(AttributeClass.NODE);
	gexf.getGraph().getAttributeLists().add(attrList);
	attrCount = attrList.createAttribute("0", AttributeType.INTEGER, "count")

	// Nodes
	node_cache = [:]
	query_tables_and_count(sql).each { t ->
		node = gexf.getGraph().createNode(t.name)
		node
			.setLabel(t.name)
			.getAttributeValues()
				.addValue(attrCount, "${t.count}")	// convert from BigDecimal
		node_cache[t.name] = node
	}

	// Relationships
	query_foreign_relationships(sql).each { r ->
		node_cache[r.key].connectTo(node_cache[r.ref])
	}

	// Write XML
	def writer = new StaxGraphWriter()
	writer.writeToStream(gexf, System.out)
}
write_xml(sql)

First I downloaded the Pidgin client plugin sourceball from the SIPE Project, and started running configure --prefix=/usr to identify missing libraries. I installed the missing dependencies using sudo apt-get install intltool libpurple comerr-dev, other systems may require more or less. After configuring, making, and installing the plugin I fired up Pidgin.

The only modification I had to make to the configuration instructions was to use SSL instead of TCP for communications as we use a certificate at my workplace. Although contacts and groups were pulled in successfully, my presence notification doesn’t seem to work (I show up as offline). However, this is good enough for me right now and I’m really happy to be able to access the corporate communication channel in my primary development environment.

*Updated to 1.6.3*
Presence notification seems to be working now, thank you SIPE team!

Controls

Left/Right Arrows or Mouse Buttons: Reverse direction or perform a step in that direction when in Step Mode.
Space or Middle Mouse Buttons: Pause and enter Step mode
Note that after the circles pass each other, the left/right controls may seem counter-intuitive.

Source

The application has two phases: generating and solving. You can restart the generation by clicking. After the maze has been generated click to begin solving from that point.

Source

# Placed in Apache conf for the site
<Proxy balancer://app_cluster >
BalancerMember http://app1/foo
BalancerMember http://app2/foo
BalancerMember http://app3/foo
</Proxy>
ProxyPass /foo/ balancer://app_cluster/ stickysession=JSESSIONID
ProxyPassReverse /foo/ http://app1/foo/
ProxyPassReverse /foo/ http://app2/foo/
ProxyPassReverse /foo/ http://app3/foo/

The lesson here is attention to detail, especially your final slashes.

I recently worked on configuring and enabling Ehcache with our Hibernate objects for common domain objects we query on almost every page request. I was interested in whether this would improve performance, and how best to illustrate the effect of the pages across threads and actions.

I had written some JMeter tests I am able to automatically run with different thread loads (5-60 threads stepping by 5) so I generated some response time data for before and after enabling the cache. Although there is actually many more rows and columns, in general the data looks like:
label elapsed threads

Action Elapsed Threads
NtSP 25 10
DLgn 11 10
FtDV 6 10
NtMM 16 10
NtSS 20 15
SrfS 157 10


I then read the log files into two data frames, r1 for the pre-cache times and r2 for the post-cache times. Using tapply I then generated two matrices of mean response times by thread and action (r1 becomes m1, r2 becomes m2). From that point it is just a matter of subtracting the m1 matrix from m2 to obtain the differences measured.

z <- r1
m1 <- tapply(z$elapsed, list(z$label, z$threads), mean)
z <- r2
m2 <- tapply(z$elapsed, list(z$label, z$threads), mean)
m.diff <- m2 - m1

Because the final output of the previous calculations is a two-dimensional matrix, a heatmap or similar visualization style seemed like it might be appropriate. In the real version the action names are not obfuscated.

# Calculate breaks using midpoints of matrix histogram with initial extension
x <- c(-20000, hist(m.diff)$mids)
levelplot(t(m.diff), scales=list(cex=0.7), aspect="iso", col.regions=heat.colors, pretty=F,at=x,xlab='Threads',ylab='Action',main="Time Difference With Cache")

metrics <- read.csv('some_metrics.csv',header=T)
aggregate(dce$elapsed, by=list(CLAZZ=dce$clazz,METHOD=dce$method), median) -> medians
aggregate(dce$elapsed, by=list(CLAZZ=dce$clazz,METHOD=dce$method), mean) -> means
aggregate(dce$elapsed, by=list(CLAZZ=dce$clazz,METHOD=dce$method), min) -> mins
aggregate(dce$elapsed, by=list(CLAZZ=dce$clazz,METHOD=dce$method), max) -> maxes
aggregate(dce$elapsed, by=list(CLAZZ=dce$clazz,METHOD=dce$method), length) -> lengths
aggregate(dce$elapsed, by=list(CLAZZ=dce$clazz,METHOD=dce$method), sum) -> sums
s <- mins
s$MIN <- s$x
s$x <- NULL
s$MAX = maxes$x
s$MEAN = means$x
s$MEDIAN = medians$x
s$NUM = lengths$x
s$SUM = sums$x
rm(mins,means,maxes,medians,sums)

This was obviously less than ideal, although I could wrap this in a function it is a bit ugly and cumbersome. I searched the R-help mailing list and found some references to the doBy package, which “grew out of a need to calculate groupwise summary statistics in a simple way”. The summaryBy function in this package turned out to be exactly what I needed and simplified by code to:

summarize <- function(csvfile) {
	require(doBy)
	metrics.csv <- read.csv(csvfile,header=T)
	metrics <- summaryBy(elapsed ~ clazz + method, data=metrics.csv, FUN=c(mean,median,min,max,sum,length))
	write.csv(metrics, file='export.csv', quote=F, row.names=F)
	metrics
}
metrics <- summarize('some_metrics.csv')

I realized it should be pretty easy to use the approach as the previous R map I generated to make a smaller scale map at a country level. In this example I’m setting up an initial non-plotted map with limits on the X and Y ranges I want to display. Next I assign colors to the observations of this subset map, plot these filled areas, then plot all boundaries.

I’m not really happy with this map because (1) I don’t really like just throwing out maps with just geographic coordinates, and (2) using area coloring can be a bit misleading about the magnitude and dispersal of the data presented: in this case there are really small clusters of reported cases, not country wide. I think this map might be a good index map to get an overview of the spread and number of cases, but would need to be backed by more detailed maps that do not generalize the location of outbreaks as much.

Here’s the source:

# load required libraries
require(maps)
require(RColorBrewer)

# Create a dataframe with the reported observations
loc <- c('USA', 'Mexico', 'Canada', 'Spain')
cases <- c(40, 26, 6, 1)
flu <- data.frame(loc,cases)

# Setup the coordinate system
m <- map('world',plot=F, xlim=c(-180,5),ylim=c(10,90), fill=T)

# Match up our observations
stm <- match.map(m, flu$loc)

# Rank the cases and assign colors using the RColorBrewer YlOrRd palette
flu$rank <- rank(flu$cases, ties='min')
pal <- brewer.pal(max(flu$rank),'YlOrRd')
color <- pal[flu$rank]
flu.color <- color[stm]

# Do the drawing
map(m,col=flu.color,fill=T, lty=0,boundary=F,interior=F) # fill regions
map('world',interior=T,add=T,col='grey30') # plot boundaries
map.axes()
grid(col='grey50')
title('WHO Confirmed Cases of Swine Flu by Country (28 April 2009)')
legend('bottomleft', legend=paste(flu$loc,flu$cases),
	fill=color, bg='white', horiz=T, cex=0.75,
	title=paste('Jason B. Smith | 28 April 2009 | Source: CNN.com'))