The Singularity

“Imagination is its own form of courage.” -Francis Underwood, House of Cards

We’re huge House of Cards fans in this household. But by huge fan, I mean we’re lazy tv watchers and we’re just about to finish season 3.

There’s been a lot of synergy going around lately. I recently got back from the ESRI UC in San Diego (AMAZING!) Out of all the workshops, vendors, talks and various other offerings, the most productive part of that (for me) were the following two things:

  • On a whim, stopping and talking to our ESRI rep in the hall.

&

  • The plenary speakers.

In House of Cards, Season 3, there’s an episode that’s driven by the tempo of Monks creating a mandala. Like all other elements of the show, nothing is done just for the sake of doing it. The metaphor is an obvious one – build something beautiful up, but nothing lasts forever. We must celebrate change, time, death and rebirth. All these things are inherently tied to each other.

The last plenary speaker at ESRI UC was Dr. Geoffrey West. He’s a theoretical physicist who’s passionate about life and death and he takes a multi-disciplinary approach to his work and has begun to apply what he’s learned to the life of cities and businesses, and consequently – their death. He compares the biological principles of life and death to our social structures and the parallelisms are striking.

He discusses “the fatal flaw” – the point of singularity. Something is built – slowly at first, and then rapidly and exponentially as time progresses (even as time speeds up – or causing time to speed up?) At some point the progression is unsustainable and the system that’s been built collapses under its own pressure. That’s the fatal flaw and the singularity. To avoid this, an innovation is necessary to “buy more time” – but because time is speeding up (or this is causing time to speed up?) the time between each necessary innovation gets shorter and shorter… the inescapable fatal flaw. The inescapable singularity of death. The beautiful mandala.

Can we innovate so profoundly that it saves us (it/whatever we build) entirely?

Depending on your mood and/or libation of choice, pondering this can lead you down some interesting rabbit holes of nihilism. If you’re eternally optimistic or have quit worrying about such things, then perhaps you’ll just enjoy the journey (I do!)

What do we really have?

Actually, moreover, what the hell does this all have to do with Geographic Information Systems? Why was this dude the plenary speaker at the ESRI UC? What are dinner parties like at Jack Dangermond’s house? SO. MANY. QUESTIONS.

If you’re paying any attention at all – which most of us aren’t – then you’ve probably figured out that the whole hippy adage “everything is connected” is actually true and that maybe patchouli doesn’t stink so bad after all. Geoffrey West was the plenary speaker because he has started to illustrate this mathematically (meta-connectedness?), and when we illustrate it mathematically, we can begin to program it… and GIS does exactly that.

Philosophically, this is much deeper, but in my day-to-day, it relates to my work. I work for a parks system and our parks system has gone through a lot of innovation lately. Wether you feel it SHOULD be this way or not, doesn’t necessarily inform the reality that it IS this way; we have leveraged the experience of being at a park to generate revenue to take care of the parks. It used to be that parks were supposed to be some altruistic thing, but parks have mostly been viewed by the history of politics as a money-sink. I don’t like it, and I don’t agree with that view, but it is reality. To survive, innovation was necessary.

I feel like we’re at an empasse again, though. Now this second innovation is coming much sooner than the first – something Dr. West discusses and illustrates. We can go on with this newfound innovation that’s proven so successful until it goes stale and we’re stuck again – or we can look at new innovations and shift the paradigm again. Enter: GIS.

What’s all of our data doing for us? Are we slaves to editing and managing? Moving files around in folders? Spell-checking names? Cleaning up unnecessary characters? Building related tables? For what? Managing data is like letting your 30 year old live in your basement rent-free. Yet every company spends a ton of money of managing data, and very little on analysis of data. That’s insane!

Make it work for its worth. These datasets are connected and they can tell us amazing things if we layer them and ask them questions we don’t even know to ask. We can see what innovation has taken place and probably discover consequences that we can leverage for new innovations. In some circles, folks call this business intelligence, but it goes well beyond that – especially when it comes to parks, conservation and the environment.

At the ESRI UC map gallery opening, I was weaving my way through the crowds when I spotted our ESRI rep. I’ve been in meetings with him before, but I’m low man on the totem pole so I’ve never spoken up. I decided to walk on by and not say anything and then – I decided that would be a huge mistake and I turned around. I interrupted his conversation (sorry not sorry!) and we started to talk. Basically I had one question: Why aren’t we doing more [where I work]? His answer was he didn’t know and he has the same question. Of course, this prompted further discussion and we decided to set up a meeting the next day to talk about this a little more “officially.”

I’ll just say – it was the most productive hour I had at the ESRI UC. We were both excited and passionate. The level of IMAGINATION is endless – you can dream up anything and the building blocks for a solution exist. Basically, I wanted to take the smart cities concept and scale it to a park or park system. Make our data work for us and make communication between systems and people seamless. EFFICIENCY. A foreign concept in government – arguably, a direct contradiction of Democracy (“Mussolini made the trains run on time.”)

It wasn’t the first time it’s been discussed. Just the first time we stuck a name on it and had a clear idea. However, it was the first time I had been given a clear idea of what infrastructure we already had, instead of being despondent thinking that we lacked the infrastructure completely!

All that to say – I feel it’s the next step. The next innovation to escape a singularity. The destruction of one mandala and the building of another. Nowe we must attempt to bring it out of theory and into practice… Currently pouring all my energy into that.

Tackling Model Builder

Model Builder intimidates the hell out of me. I think once, a while back, I heard the words “python” and “script” and decided I’d never wander over yonder to them dark hills. So every process I do is done painstakingly by hand one-by-one until the final result is achieved…

For the most part, this has worked just fine for me. I’ve never had to be efficient about my workflow and no one has ever asked questions. So I slink around in my non-model-builder shame without exposure. This results in a lot of trash files, but it’s also resulted in knowing the functionality of these tools intimately well. I’ve always felt that learning something the hard way is the best way to learn… or at least that seems to be how I’ve stumbled through this life so far!

Now, though – I’m expected to know these things and the work load requires that many of my slow processes become more efficient and automated. So – I had to acquaint myself with model builder and I’ve been doing just that. I’m pretty happy that as of today, I’ve built my first complete tool and I’m really happy with the result.

The setup: We keep a layer that contains all our agency-owned lands by parcel/transaction. This layer really serves an internal purpose of tracking those transactions and then a secondary (but equal) purpose of keeping our boundaries up-to-date. Visually, though, it’s unpleasing. Likewise, the attribute table is full of fields that have no cartographic purpose and mean nothing to the public. The boundaries are also derived from various sources with various levels of accuracy, so slivers and gaps abound, yet legitimate inholdings should be preserved. We need to publish a clean, pretty layer for public use (as well as our own cartographic use) once a month.

Problem: Dissolve tool doesn’t cut it. It still leaves messy slivers. Aggregate takes care of this, but it totally wipes the attribute table. A spatial join can put Humpty Dumpty back together again, but it attaches ALL the attributes when we just want a couple. Some of the lands aren’t necessarily open to the public ( or part of a dataset that we ultimately want to publish), so there’s a query needed at some point to filter these. Maintaining separate datasets for each scenario sounds like a nightmare. Slivers need to be handled with care because we don’t want to exclude legitimate inholdings and make landowners angry at us.

Solution: I needed to build a tool that allows me to query what I’m interested in “dissolving”, dissolves it, takes care of slivers but MAINTAINS inholdings with aggregate, then spatially joins an interim layer from the original query to attach the park name. I’ve wrestled with this model for a couple weeks now. At once, I’m teaching myself how to use model builder and also have a practical end product. I finally got an output that is (mostly) what I’m looking for. Of course, this works on the assumption that your original layer is beautiful and organized. In our case – that is a work in progress (but close to being finished!)

The before and after:

An example of what the surveys layer looks like prior to processing. This is all part of one park, but composed of multiple acquisitions over the years.
“Dissolved” result – visually. Attribute table also cleaned up! Shows only the park name and none of the boring academic stuff for internal use.

 

 

 

 

 

 

 

 

 

This is what the layout looks like in model builder. Basically, I‘m so fancy:

Behold! My first model!

This is what the tool likes like when run from toolbox:

The simple prompt – showing SQL. Also pretty proud of my help text… yeaaaaa.

Lastly – here’s the fancy python script:

# -*- coding: utf-8 -*-
# ---------------------------------------------------------------------------
# PrettyParkPolys.py
# Created on: 2017-07-05 16:14:40.00000
# (generated by ArcGIS/ModelBuilder)
# Usage: PrettyParkPolys 
# Description:
# This tool takes the surveys layer and dissolves them into nice, clean polygons. Then that output is spatially joined to re-assign their names.
# ---------------------------------------------------------------------------

# Set the necessary product code
# import arcinfo

# Import arcpy module
import arcpy

# Script arguments
Input_Features = arcpy.GetParameterAsText(0)

Output_Lands = arcpy.GetParameterAsText(1)

Expression = arcpy.GetParameterAsText(2)
if Expression == '#' or not Expression:
Expression = "Management = 'TDEC - TSP'" # provide a default value if unspecified

# Local variables:
Lands_Layer = "Lands_Layer"
Preserve_orthogonal_shape = "true"
Transfer_field_domain_descriptions = "false"
Maintain_fully_qualified_field_names = "true"
DissolveTest_Tbl = "E:\\TDEC\\ToolExperiment\\DissolveTest_Tbl"
Aggregated_Output = "E:\\TDEC\\ToolExperiment\\DissolveTest.shp"
Integrated_Aggregated_Output = Aggregated_Output
XY_Tolerance = "20 Feet"

# Process: Make Feature Layer
arcpy.MakeFeatureLayer_management(Input_Features, Lands_Layer, Expression, "", "OBJECTID OBJECTID VISIBLE NONE;Shape Shape HIDDEN NONE;PARK_MANGE PARK_MANGE HIDDEN NONE;DATE_ACQUI DATE_ACQUI HIDDEN NONE;BND_ORIGIN BND_ORIGIN HIDDEN NONE;TAG TAG HIDDEN NONE;SECOND_NA SECOND_NA HIDDEN NONE;UPDATE_ UPDATE_ HIDDEN NONE;UPDATE_DATE UPDATE_DATE HIDDEN NONE;COMMENT_ COMMENT_ HIDDEN NONE;OWNER OWNER HIDDEN NONE;PUBLIC_ PUBLIC_ HIDDEN NONE;TRANSACTION_NO_ TRANSACTION_NO_ HIDDEN NONE;REGION REGION HIDDEN NONE;DEEDACRES DEEDACRES HIDDEN NONE;Shape_Length Shape_Length HIDDEN NONE;Shape_Area Shape_Area HIDDEN NONE;Ownership Ownership HIDDEN NONE;Bndry_Origin Bndry_Origin HIDDEN NONE;Prim_Desig Prim_Desig HIDDEN NONE;Sec_Desig Sec_Desig HIDDEN NONE;Tert_Desig Tert_Desig HIDDEN NONE;Access Access VISIBLE NONE;Management Management HIDDEN NONE;APN APN HIDDEN NONE;PRIMARY PRIMARY VISIBLE NONE;Secondary Secondary VISIBLE NONE;Calc_Acres Calc_Acres VISIBLE NONE")

# Process: Aggregate Polygons
tempEnvironment0 = arcpy.env.transferDomains
arcpy.env.transferDomains = Transfer field domain descriptions
tempEnvironment1 = arcpy.env.qualifiedFieldNames
arcpy.env.qualifiedFieldNames = Maintain fully qualified field names
arcpy.AggregatePolygons_cartography(Lands_Layer, Aggregated_Output, "20 Meters", "0 SquareFeet", "0 SquareFeet", Preserve_orthogonal_shape, "", DissolveTest_Tbl)
arcpy.env.transferDomains = tempEnvironment0
arcpy.env.qualifiedFieldNames = tempEnvironment1

# Process: Integrate
arcpy.Integrate_management("E:\\TDEC\\ToolExperiment\\DissolveTest.shp #", XY_Tolerance)

# Process: Spatial Join
tempEnvironment0 = arcpy.env.qualifiedFieldNames
arcpy.env.qualifiedFieldNames = "true"
arcpy.SpatialJoin_analysis(Aggregated_Output, Lands_Layer, Output_Lands, "JOIN_ONE_TO_ONE", "KEEP_COMMON", "Id \"Id\" true true false 6 Long 0 6 ,First,#,E:\\TDEC\\ToolExperiment\\DissolveTest.shp,Id,-1,-1;Access \"Access\" true true false 50 Text 0 0 ,First,#,Lands_Layer,Access,-1,-1;PRIMARY \"PRIMARY\" true true false 70 Text 0 0 ,First,#,Lands_Layer,PRIMARY,-1,-1;Secondary \"Secondary\" true true false 70 Text 0 0 ,First,#,Lands_Layer,Secondary,-1,-1;Calc_Acres \"Calc_Acres\" true true false 8 Double 0 0 ,First,#,Lands_Layer,Calc_Acres,-1,-1", "INTERSECT", "20 Meters", "")
arcpy.env.qualifiedFieldNames = tempEnvironment0

Harvesting … data

Gardening is, for me at least, a hobby for an introvert. In the south where the heat is sweltering and the humidity makes it that much worse – plants really like to grow if they aren’t managed into submission. Gardening is a full time job. This is usually great for me! I’m a home body and while everyone in Nashville seems to be enamored by the music and bar scene, I had enough of that growing up here, so I stay at home and mindlessly pull weeds.

Lately – I’ve not been this person. I’ve been a social butterfly and it’s been a nice refreshing change, except that all the things at home that I love to do when I’m an introvert are now totally out of hand. The garden is going crazy and the vegetables have bolted or over-ripened. C is getting suspicious of my late-night-galavanting, and the dogs are wondering who this stranger is in their house.

So it goes with data at work too… updating and sharing data with other agencies and entities is always this cumbersome and time-consuming thing. Who did you share data with, again? Who needs an updated layer? Did I respond to joe-schmo’s email requesting, again, all those shapefiles? Did I remember to strip out all the fields that are worthless to others? Keeping the GIS garden weeded is actually a major part of my full time job. It’s not a unique problem. Unless you’re a stooge who works in a dark basement making maps only for yourself, then the likelihood that you have a need to receive, create and share data is probably really high. Not only do we need to make sure others have our data up to date, but we need to make sure the data we use from them is also up to date. Two way street. This is called harvesting data. (I’m sorry – but that term just reminds me of black-market horror stories of illegally harvested body parts or something. Ew.)

ESRI has come up with a solution (albeit imperfect) for helping the data junkies keep our data gardens nice and tidy while sharing and harvesting. It’s called Open Data Portal and it’s basically like WordPress for GIS users… especially GIS users with no developer skills (ME!)1

What’s the purpose of Open Data Portal? Well – I can update my layer on desktop, have it publish as a feature service on AGOL at some interval I may set with a script, and that will update the layer I point to and send it out on Open Data Portal. Other agencies can add my ODP to theirs and so whatever data they may consistently ask for from me will just always be beautifully updated and in one spot. If someone is looking for trails and they search for state parks, they will find trails through a state parks page. If they’re looking for trails but they happen to look for it through the state of Tennessee, then they can find the exact same updated layer through another related agency who didn’t necessarily author it. The authorship carries along with it through ODP! It’s fun and means I don’t have to think about who to contact to tell them they need an updated layer or I need theirs. It also exponentially increases the exposure of your data to the public. FREE MARKETING.

So – I was volunteered/did volunteer to take on experimenting with this new platform for my little slice-of-the-state-pie agency.2 I ain’t scared. I’ll experiment with anything. So this week I finally tackled it full on and here’s some quick observations.

1.) The idea is really elegant, but the execution is not.

It’s like a really really shitty version of WordPress. Laying out your page that serves as the “portal” is a drag-and-drop type of CMS. That’s all great and good, except the functionality is super limited and so I’m wrestling with some jacked up markdown and HTML to try and get things to behave the way I want them to behave. For example – if I drag a box layout for an image, then I’d really like to have the option to attach HTML to that image to link to one of the sub-pages of the portal site. NOPE. I have to drag the TEXT box layout and then type a bunch of gobbly-goop in there to get that to work and I’ll have to do that for every page I want to point to! Annoying. Drag-and-drop fail.

2.) What a “group” is and how it functions as it relates to classic AGOL and ODP are pretty different – so perhaps we shouldn’t call them both groups?

ESRI hasn’t really re-imagined their nomenclature on this much. A group in the classic AGOL environment is set up like a little club to share/receive/edit data with a restricted group of people. A group for Open Data Portal (ODP) suddenly has different functionality. It’s for the above, but now it’s also for adding data TO the group so that it points TO the portal. This is kind of annoying and calling both functions “groups” can get confusing. It’s not at all intuitive.

3.) Sites vs. Pages.

ESRI- I’m sorry, dear. I’m your biggest fan girl but holy crap! If you’re going to give us the ability to make pages under a larger umbrella site then WHY OH WHY did you not also come up with an easy way to put your pages as tabs or drop down or whatever on said site?! Pages is an awesome idea, but it’s like you got sidetracked and quit working on it. LET ME DRAG AND DROP PAGES AS PART OF THE LAYOUT DRAG AND DROP STUFF! That seems so obvious and yet…

3.) Homogenization problems

I added an academic institution to our ODP today. When you add another ODP to your ODP then all the layers that they’ve already added to their ODP shows up in yours! Confused yet? It’s like those terrifying charts they showed you in sex ed that illustrate how, if you’ve slept with one person, then you’ve slept with an exponential number of people simply by association. So now all of a sudden in my agencies ODP we have layers that are totally unrelated to something someone might land on our page for. Kind of cool – but – I can see this homogenizing datasets and actually making Open Data Portal kind of worthless and useless. What’s the point of having my own subset page if anything and everything from anyone I’ve ever added is on it?  Not a fan – but I suspect more flexible functionality for what one would like to hide/show is coming in the future.

But really – I’m a huge ESRI fan girl so it would take Jack Dangermond skinning baby seals or something for me to drop them totally. Even then- it’s like hating your favorite band because the lead singer is a jerk. I just don’t roll like that.

Bottom line: Open Data Portal is new. It’s an awesome idea. I’m super psyched about it but right now there are clearly some growing pains. As with all other ESRI products, I’m willing to suffer through those pains until it becomes awesome.

1 Apparently folks also use CKAN which seems way beyond my abilities, but I’m intrigued.

2 When we get ODP up and running, links will be posted. Duh.

In the Beginning…

The beginnings and endings of all human undertaking are untidy. ~John Galsworthy from some book no one has read, including me.

I didn’t used to be one for plans. I don’t know what’s changed… lacking a plan always seemed to work out for me splendidly. As we get older though, I think most of us gain a sense of mortality and we quit climbing the big trees.

I envy those that lack that mortal sense, but I’m inspired by that distant version of myself that still lingers in my heart. The person who had no plan and no cares and went along with wherever the world was going to have me.

GIS began as an accident. I had transferred from one university to another during my undergrad and in between I spent my time as an ecology and botany intern at an air force base. It was chock full of Coastal Plain disjuncts and heliophytic 1 species indicative of historic barrens and prairie. I knew nothing then, botanically, but my previous professor had encouraged me to take the position and the pay was excellent! (Note: my boss at the time told me that I would never make that much money again in my career. Note 2: I’m a highly competitive person and took that as a personal challenge.)

While I was an intern there I had heard about this GIS business and asked one day if I could learn a little bit about it. I recall them setting me up on modules for learning where the buttons were in Arc 9. I made a hot pink map with black and white tracks with skulls for waypoints. Man! I wish I had that map to put here. I’ll have to re-create it.

Fast forward about a year and I’m in the botany lab at UT-Chattanooga. My adviser was working on a giant proposal for a corridor study. The goal was to study and map the flora and floristic communities through a river gorge with highly unstable geology. Landslides kept happening along the road that was (and is) a major corridor for the region. I was hired as the lowly undergrad to press specimens and retrieve coffee.

Except one pitfall- no one knew really how to map it all. Sensing an opportunity I told my adviser

“Hey – I’ve done GIS when I was an intern. Let me map it. We’ll use it as Plan B and if we get someone who actually knows what they’re doing, we’ll go with their work. What’s to lose?”

They said ok. THEY SAID OK.

Fast forward four years and I’ve cut back my credit load to part time and I’m spending most of my time in the botany lab on a sketchy computer managing a quarter-of-a-million-dollar project, orchestrating and organizing the grad students, the collecting trips, the specimen pressing/drying and of course, mapping it all. It was a hardcore crash course in GIS and I was totally addicted. I lived it and breathed it and I owned that project. It was my baby and I loved it. The field work was amazing – some of these places hadn’t been seen since Augustin Gattinger 2 in the 1800’s. The views were incredible, the river was gorgeous, the need for this study was real and practical and I felt like I was really making a difference. With the GIS – it was a wrestling match. It was my favorite boyfriend. It rewarded me and fulfilled me, but it perplexed me and kept me on my toes and taught me so much. I was hooked and obsessed. It was niche. It was mine.

The product was passable – it went into the reports and the client, of course, got all the data. I laugh when I see the maps now. THEY WERE TERRIBLE. I literally drew every little polygon by hand. I was totally clueless. I had no real idea how to use raster data, how to edit en-masse, how to use model builder to help my workflows. I would set a geoprocess on the computer and let it run through lunch, or overnight. It was SO SLOW. I had no idea. No clue.

and neither did anyone else…

So that’s how it all began. Check out this sketchy stuff:

My favorite part of that is that I was actually comparing how much “better” I got over those two years. I started the project in 2010. We ended the project in 2013. If I could do it all over again, I totally would.

Lessons learned:

  • -I learned a ton of GIS, obviously.
  • -If you’re working on a project and you want to involve GIS at all, then you need to begin the project FROM the perspective of GIS (which brings me to the third point)
  • -Data is EVERYTHINGIf you’re having to collect data for a project, then you better think about that from the very nebulus of your project’s beginning. How is it going to work in your geodatabase? How do you need it to be structured? What question are you trying to answer and how are you going to answer it? This really goes beyond data structure. It involves your licensing, your processing, your budget. Everything.
  • Most importantly, I learned…
  • I CAN TEACH MYSELF ANYTHING

I’ve come a hell of a long ways since then and I still have a long ways to go, but I felt like I had wrestled some colossal beast and tamed it. Even though the GIS was shoddy at best, I continue to be the most proud of this project.

Heliophyte. Helio = sun. Phyte = loving. Sun-loving plants. Need open habitats to thrive.
2 Augustin Gattinger was a German ex-pat. He didn’t really like Tennessee and he was kind of disgruntled, but he was a super badass and amazing botanist and you should read more about him.

Debut

The idea of being pigeon-holed into doing one thing the rest of my life terrifies me.  (That’s the confessional aspect of the blog out of the way right off the bat, yes?)

Point being – “GIS Specialist” is at once an awesome title, and a misnomer. I’m a “GIS generalist” and that means I get to work on a wide range of GIS projects that most of my colleagues don’t get to.

Upside- I’m never bored.

Downside – I’m a jack of all trades, and master of none.

As the first blog should really set the tone, we’re just all going to sit on the GIS therapy couch and get a few things out of the way right off the bat… Set the framework, so-to-speak:

1.) I’m entirely self-taught

2.) My background is botany and ecology.

3.) I’m a terrible botanist, but a pretty good ecologist, and probably best at GIS.

4.) I love experimenting. (In life, really, but I’ll keep it kosher here… mostly.)

Due to the aforementioned, some of my workflows are entirely experimental; pieced together like a frankenstein and set forth in this world without a care. I’ll be using this blog to chronicle the brilliant moments, the terrible fails, the products that derive from it all, and other random musings. Feedback from other GIS users is most-appreciated.