An algorithm must be seen to be believed.

~ D. Knuth

First i trust everyone is safe. Second its WEDNESDAY so we got us a Snake_Byte! Today i wanted to keep this simple, fun and return to a set of fun methods that are included in the defacto standard for plotting in python which is Matplotlib. The method(s) are called XKCD Style plotting via plt.xkcd().

If you don’t know what is referencing it is xkcd, sometimes styled XKCD, whcih is a webcomic created in 2005 by American author Randall Munroe. The comic’s tagline describes it as “a webcomic of romance, sarcasm, math, and language”. Munroe states on the comic’s website that the name of the comic is not an initialism but “just a word with no phonetic pronunciation”. i personally have read it since its inception in 2005. The creativity is astounding.

Which brings us to the current Snake_Byte. If you want to have some fun and creativity in your marketechure[1] and spend fewer hours on those power points bust out some plt.xkcd() style plots!

First thing is you need to install matplotlib:

pip install matplotlib

in this simple example we need numpy:

pip install numpy
import numpy as np
plt.xkcd()
plt.plot(np.sin(np.linspace(0, 10)))
plt.plot(np.sin(np.linspace(10, 20)))
plt.title('Sorta Lissajous')

So really that is all there with all the bells and whistles that matplotlib has to offer.

The following script was based on Randall Munroe’s Stove Ownership.

(Some will get the inside industry joke.)

with plt.xkcd():
# Based on "Stove Ownership" from XKCD by Randall Munroe
# https://xkcd.com/418/

fig = plt.figure()
ax = fig.add_axes((0.1, 0.2, 0.8, 0.7))
ax.spines.right.set_color('none')
ax.spines.top.set_color('none')
ax.set_xticks([])
ax.set_yticks([])
ax.set_ylim([-30, 10])

data = np.ones(100)
data[70:] -= np.arange(30)

ax.annotate(
'THE DAY I TRIED TO CREATE \nAN INTEROPERABLE SOLTUION\nIN HEALTH IT',
xy=(70, 1), arrowprops=dict(arrowstyle='->'), xytext=(15, -10))

ax.plot(data)

ax.set_xlabel('time')
ax.set_ylabel('MY OVERALL MENTAL SANITY')
fig.text(
0.5, 0.05,
'"Stove Ownership" from xkcd by Randall Munroe',
ha='center')

So dear readers there it is an oldie but goodie and it is so flexible! Add it to your slideware or maretechure or just add it because its cool.

Until Then,

#iwishyouwater <- Mentawis surfing paradise. At least someone is living.

Muzak To Blog By: NULL

[1] Marchitecture is a portmanteau of the words marketing and architecture. The term is applied to any form of electronic architecture perceived to have been produced purely for marketing reasons and has in many companies replaced actual software creation.

The thing that you are most comfortable with that you do the best.

Steve Vai

First as always i hope everyone is safe. Second this blog is out of band so to speak. This question was posited to me during a technical discussion with some great folks whom i had just met and we were discussing re-tooling and scaling enterprise systems.

Completely almost non-sequitur this executive asked:

Q: What do you think you are good at?

There isnt too many times when i am personally caught off gaurd but i stopped and replied “Well that is a great question. So good in fact i am going to write a blog if you dont mind. “

Which then got me to thinking:

Q: If you really truly love what you are doing is it really work?

Not at all.

This goes along with several of the blogs i have done in the past concerning – What Is IT You Truly Want?

So without any hubris or narcissism as far as i know or have been told what i am good at is the following:

Evidently, i have an uncanny ability to see what needs to be built with the right team and at a pretty good time or within a certain timeframe.

The other thing i am supposedly good at is getting people aligned and excited around a common vision to execute said code base or system(s).

i also take the title CTO very seriously. i do pride myself on keeping up with technology. i attempt to find out what works and what does not work. Mathematics and Software completely and unequivocally enthrall me. i am always either reading a book, paper or blog on a technical subject. i don’t really keep up with the normal outside world. So i can’t really comment on sports, movies, or the daily news. Ergo i don’t go around chasing fads however many times you do have to create net new warez. i tend to go deep on technical subjects before i bring them into an organization.

Really after thinking about all of this is it useful? After thinking more about it i really cannot say at this time. Maybe it’s an occupational hazard nowadays. However, it is what i maybe think i am good at so to speak.

However, i can assure you that holding that mirror up to yourself and looking deep into it is an exercise we should all do on a daily basis. As the famous song lyric goes “Chickity-check yo’ self before you wreck yo’ self”.

So finding out what you are really good at and strengthening that creates a self-perpetuating system. It has been said You Are Your Best Charity. If you truly enjoy what you are doing then it really isn’t work is it? This allows you to concentrate on what you are good at and then in turn Amplifying_OthersTM.

Find Your Passion At All Costs.

Become the system You are creating.

Then IT will naturally happen.

I posted this video of Steve Vai a long time ago in another blog and in another life. i ran across it while taking a break at 2:30 AM EST working on a very serious bug with a company i co-founded. It paused me. i always come back to it. While this is supposedly a master class with Steve Vai he never talks about guitar technique but 1rather how to be successful (really at anything).

Possibly the only thing that i think is more amazing than creating software is music creation. Here is Mr. Steve Vai doing what he does best. Do yourself a favor, watch and listen. i’ll hopefully be seeing him soon in Charleston, SC.

Would love some comments and feedback on this blog. While it is short i have to tell you it was difficult to type those couple of sentences.

Until Then,

#iwishyouwater <- Will Trubridge 60M Freedive in 60 seconds

@tctjr

Muzak To Blog By: Steve Vai.

Software Is The Language Of Automation

Jensen Huang

First, i trust everyone is safe. Second: Hey Now! Wednesday is already here again! Why did Willy Wonka say about “So Much Time And So Little To Do?” Or better yet “Time Is Fun When You Are Having Flies!” Snake_Byte[8] Time!

This is a serendipitous one because i stumbled onto a library that uses a library that i mentioned in my last Snake_Bytes which was pandas. It’s called MitoSheets and it auto-generates code for your data wrangling needs and also allows you to configure and graph within your Jupyter_Lab_Notebooks. i was skeptical.

So we will start at the beginning which is where most things start:

i am making the assumption you are either using a venv or conda etc. i use a venv so here are the installation steps:

pip install mitoinstaller
pip mitoinstaller install

Note the two step process you need both to instantiate the entire library.

Next crank up ye ole Jupyter Lab:

import mitosheets
mito.sheet()

It throws up a wonky splash screen to grab your digits and email to push you information on the Pro_version i imagine.

Then you can select a file. i went with the nba.csv file from the last blog Snake_Bytes[7] Pandas Not The Animal. Find it here :

Then low and behold it spit out the following code:

from mitosheet import *; register_analysis("id-ydobpddcec");

# Imported nba.csv
import pandas as pd
nba = pd.read_csv(r'nba.csv')

register_analysis("id-ydobpddcec") is locked to the respective file.

So how easy is it to graph? Well, it was trivial. Select graph then X & Y axis:

So naturally i wanted to change the graph to purple and add some grid lines with a legend to test the export and here was the result:

As Henry Ford said you can have any color car as long as it is black. In this case you are stock with the above graph while useful it’s not going to catch anyone’s eye.

Then i tried to create a pivot table and it spit out the following code:

from mitosheet import *; register_analysis("id-ydobpddcec");

# Imported nba.csv
import pandas as pd

# Pivoted into nba
tmp_df = nba[['Team', 'Position', 'Number']]
pivot_table = tmp_df.pivot_table(
index=['Team'],
columns=['Number'],
values=['Position'],
aggfunc={'Position': ['count']}
)
pivot_table.set_axis([flatten_column_header(col) for col in pivot_table.keys()], axis=1, inplace=True)
nba_pivot = pivot_table.reset_index()

Note the judicious use of our friend the pandas library.

Changing the datatype is easy:

from mitosheet import *; register_analysis("id-ydobpddcec");

# Imported nba.csv
import pandas as pd

# Changed Salary to dtype datetime
import pandas as pd
nba['Salary'] = pd.to_datetime(nba['Salary'], unit='s', errors='coerce')


It also lets you clear the current analysis:

So i started experimenting with the filtering:

from mitosheet import *; register_analysis("id-ydobpddcec");

# Imported nba.csv
import pandas as pd

# Filtered Weight
nba = nba[nba['Weight'] < 180]

The views for modification are on the right side of the layout of the table which is very convenient. The automatic statistics and visualizations are helpful as well:

The max,min,median, and std are very useful and thoughtful:

The following in and of itself could be enough to pip install the library:

You can even have multiple dataframes that can be merged. Between those items and the summary stats for those that are experienced this could be enough price to entry to pip install and then install the library. For those that really don’t know how to code this allows you to copypasta code and learn some pretty basic yet very powerful immediate insights into data. Also if you are a business analyst, a developer could get you going in no time with this library.

i don’t particularly like the lockouts on the paywall for features. In today’s age of open-source humans will get around that issue and just use something else, especially the experienced folks. However, what caught my attention was the formatting and immediate results with a code base that is useful elsewhere, so i think the Mito developer team is headed in the right direction. i really can see this library evolving and adding sklearn and who knows Github Copilot. Good on them.

Give it a test drive.

Until Then,

#iwishyouwater <- #OuterKnown Tahiti Pro 2022 – Best Waves

@tctjr

Muzak To Blog By: Tracks from “Joe’s Garage” by Frank Zappa. “A Little Green Rosetta” is hilarious as well as a testament to Zappa’s ability to put together truly astound musicians. i love the central scrutinizer and “Watermelon in Easter Hey” i believe is one of the best guitar pieces of all time. Even Zappa said it was one of his best pieces and to this day Dweezil Zappa is the only person allowed to play it. One of my readers when i reviewed the Zappa documentary called the piece “intoxicating”. Another exciting aspect of this album is that he used live guitar solos and dubbed them into the studio work except for Watermelon In Easter Hey. The other Muzak was by a band that put Atlanta on the map: Outkast. SpeakerBoxx is phenomenal and Andre3000 is an amazing musician. “Prototype” and “Pink & Blue”. Wew.

We are organized like a startup. We are the biggest startup on the planet.

S. Jobs

First, i hope everyone is safe. Second, this blog is about something everyone seems to be asking me about and talking about, but no one seems to be able to execute the concept much like interoperability in #HealthIT. Third, it is long-form content so in most cases tl;dr.

CULTURE.

Let us look to Miriam-Websters OnLine Dictionary for a definition – shall we?

cul·​ture <ˈkəl-chər>

1

a: the customary beliefs, social forms, and material traits of a racial, religious, or social group also the characteristic features of everyday existence (such as diversions or a way of life) shared by people in a place or time ; popular culture, Southern culture

b: the set of shared attitudes, values, goals, and practices that characterizes an institution or organization a corporate culture focused on the bottom line

c: the set of values, conventions, or social practices associated with a particular field, activity, or societal characteristic studying the effect of computers on print culture

d: the integrated pattern of human knowledge, belief, and behavior that depends upon the capacity for learning and transmitting knowledge to succeeding generations

2

a: enlightenment and excellence of taste acquired by intellectual and aesthetic training

b: acquaintance with and taste in fine arts, humanities, and broad aspects of science as distinguished from vocational and technical skills; a person of culture

3: the act or process of cultivating living material (such as bacteria or viruses) in prepared nutrient media also a product of such cultivation

5: the act of developing the intellectual and moral faculties especially by education

6: expert care and training; beauty culture

Wow, This sounds complicated. Which one to leave in and which one to leave out?

Add to this complexity the fact that creating and executing production software is almost an insurmountable task. i have said for years software creation is one of the most significant human endeavors of all time. i also believe related to these concerns the interplay between comfort and solutions. Most if not all humans desire solutions however as far as i can tell solutions are never comfortable. Solutions involve change most humans are homeostatic. Juxtapose this against the fact that humans love comfort. So what do you do?

So why does it seem like everyone is talking about kəl-chər? i consider this to be like Fight Club. 1st rule of kəl-chər is you don’t talk about culture. It should be an implicit aspect of your organization. Build or Re-Build it at a first principles engineering practice. Perform root cause analysis of the behaviors within the company. If it does in fact need to be re-built start with you and your leadership. Turn the mirror on you first. Understand that you must lead by example. Merit Not Inherit.

i’ve recently been asked how you change and align culture. Well here are my recommendations and it comes down to TRUST at ALL levels of the company.

Create an I3 Lab: Innovation, Incubation, Intrapreneurship:

Innovation without code is just ideas and everyone has them. Ideas are cheap. Incubation without product market fit is a dead code base. Intrapreneurship is the spirit of a system that encourages employees to think and act like individual entrepreneurs and empowers them to take action, embrace risk, and make decisions as if they had founded the company themselves. Innovate – create the idea – Incubate – create the Maximum Viable Product (not minimum) – Intrapreneurship – spin out the Maximum Viable Product. As an aside Minimum Viable Product sounds like you bailed out making the best you possibly could in the moment. Take that Maximum Viable product and roll it into a business vertical and go to market strategy – then spin the wheel again.

I think it’s very important to have a feedback loop, where you’re constantly thinking about what you’ve done and how you could be doing it better.

E. Musk

Value The Most Important Asset – Your People

Managing high-performance humans is a difficult task because most high-performance humans do not like to be managed they love to be led. Lead them by example. Value them and compensate them accordingly. Knowledge workers love achievement and goals. Lead them into the impossible, gravitate toward dizzying heights, and be there for them. Be completely transparent and communicate. Software is always broken. If anyone states differently they are not telling the truth. There is always refactoring, retargeting, more code coverage and nascent bugs. Let them realize you realize that however let them know that if they do make a mistake escalate immediately. Under no circumstances can you tolerate surprises. Give them the framework with OKRs and KPIs that let them communicate openly, efficiently and effectively and most important transparently. Great teams will turn pencils into Mount Blanc Fountain Pens. Let them do what they do best and reward them!

Process Doesn’t Make A Culture

Nor does it make great products. Many focus on some software process. Apple used and as far as i know still uses strict waterfall. As far as i am concerned, we are now trending towards a Holacracy type of environment which is a self-organizing environment. However, this only can be achieved with the proper folks that appreciate the friction of creating great products from the best ideas. The Process of evolving from an idea to a product is magic. You learn you evolve; you grow your passion for and into the product as it becomes itself the team that built the product. Your idea and passion are inherent in that shipping software (or hardware).

What do you want me to do

To do for you to see you through?

Provide your people the ability to manage themselves and have autonomy. Set them free. Trust them to make the decisions that will drive the company and projects into world-class endeavors. Take a chance with them, Let a new college graduate push some code to production. Let a new sales associate push a deal with a customer. Let your new marketing person design an area on the company site. Allow them to evolve grow and be a part of the Great Endeavor. Put them in charge and provide the framework for autonomy to make decisions and when they deliver – award them not with something ephemeral but volitional. Money and Stock work wonders. Empower. Align. Evolve.

Provide and Articulate a Common Vision

Provide a vision of the company or project. Two sentences that everyone understands. Most people who are empowered and given the frameworks to create within know what to do in these circumstances. Articulate the vision and gain common alignment across the organization or project. That is leadership that high performance teams desire. Take this alignment then map it into the OKRs and KPIs then in turn pick a process and let everyone know how this aligns to the vision. Create the environment that every line of code maps to that vision. Show commitment on this vision.

Give FeedBack

Communicate. Communicate. Communicate. Collaborate. Collaborate. Collaborate. Till you puke. i cannot emphasize this enough. You must be prepared everyday to manically interact with your teams and have the hard friction filled uncomfortable discussions. You want to keep the top performers let them know where they stand, how they stand and why they stand in the rankings and how they are contributing to the vision. Again attempt to create coder-metrics across your organization or project that exemplifies this performance. Interact with your most important asset your people. Over communicate. We have the ability to reach everyone at anytime, email, zoom, slack, granite tablet where once used to message. Write the message and give feedback. Better yet go take a walk with them. Have 1:1s. Listen to your people receptively and without bias and judgment about their concerns, passions, what scares them, what makes them happy, their joys, goals, and aspirations so they feel validated and understood. Solicit feedback, shut up and listen.

What all of this comes down to what i call – Amplifying_OthersTM. This is easier said than done. Personally, i believe that you need to commit even to the point of possibly finding them a better fit for a position at another company. This goes back to understanding what truly drives the only asset there is in technology the people. Always Be Listening, Always Be Networking, and Always Be Recruiting.

This brings up the next big question for your company – How do you attract the best right talent? Hmmmm… that might be another blog. Let me know your thought on these matters in the comments.

Until Then,

#IWishYouWater <- Psycho Session In Mentawis

@tctjr

Music To Blog By:

American Beauty by The Grateful Dead. Box of Rain and Ripple are amazing. Also if you haven’t heard Jane’s Addiction’s cover of Ripple check it out. i am not a Dead fan but the lyrics on some of these songs are monumental.

The Psychology of Computer Programming

Mythical Man Month

The Essence of Software: Why Concepts Matter for Great Design

DISCLAIMER: This blog was written some time ago. Software breaks once in a while and there was a ghost in my LazyWebTM machine. We are back to our regularly scheduled program. Read on Dear Reader, and your humble narrator apologizes.

The other day i was talking to someone about file manipulations and whatnot and mentioned how awesome Pandas and the magic of the df.DoWhatEverYaWant( my_data_object) via a dataframe was and they weren’t really familiar with Pandas. So being that no one knows everything i figured i would write a Snake_Byte[] about Pandas. i believe i met the author of pandas – Wes Mckinney at a PyData conference years ago at Facebook. Really nice human and has created one of the most used libraries for data wrangling.

One of the most nagging issues with machine learning, in general, is the access of high integrity canonical training sets or even just high integrity data sets writ large.

By my estimate over the years having performed various types of learning systems and algorithm development, machine learning is 80% data preparation, 10% data piping, 5% training, and 5% banging your head against the keyboard. Caveat Emptor – variable rates apply, depending on the industry vertical.

It is well known that there are basically three main attributes to the integrity of the data: complete, atomic, and well-annotated.

Complete data sets mean analytical results for all required influent and effluent constituents as specified in the effluent standard for a specific site on a specific date.

Atomic data sets are data elements that represent the lowest level of detail. For example, in a daily sales report, the individual items that are sold are atomic data, whereas roll-ups such as invoices and summary totals from invoices are aggregate data.

Well-annotated data sets are the categorization and labeling of data for ML applications. Training data must be properly categorized and annotated for a specific use case. With high-quality, human-powered data annotation, companies can build and improve ML implementations. This is where we get into issues such as Gold Standard Sets and Provenance of Data.

Installing Pandas:

Note: Before you install Pandas, you must bear in mind that it supports only Python versions 3.7, 3.8, and 3.9.

I am also assuming you are using some type of virtual environment.

As per the usual installation packages for most Python libraries:

pip install pandas

You can also choose to use a package manager in which case it’s probably already included.

#import pandas pd is the industry shorthand
import pandas as pd
#check the version
pd.__version__
[2]: '1.4.3'

Ok we have it set up correctly.

So what is pandas?

Glad you asked, i have always thought of pandas as enhancing numpy as pandas is built on numpy. numpy It is the fundamental library of python, used to perform scientific computing. It provides high-performance multidimensional arrays and tools to deal with them. A numPy array is a grid of values (of the same type) indexed by a tuple of positive integers, numpy arrays are fast, easy to understand, and give users the right to perform calculations across arrays. pandas on the other hand provides high-performance, fast, easy-to-use data structures, and data analysis tools for manipulating numeric data and most importantly time series manipulation.

So lets start with the pandas series object which is a one dimensional array of indexed data which can be created from a list or an array:

data = pd.Series([0.1,0.2,0.3,0.4, 0.5])
data
[5]: 0    0.1
1    0.2
2    0.3
3    0.4
4    0.5
dtype: float64

The cool thing about this output is that Series creates and wraps both a sequence and the related indices; ergo we can access both the values and index attributes. To double check this we can access values:

[6]: data.values
[6]: array([0.1, 0.2, 0.3, 0.4, 0.5])

and the index:

[7]: data.index
[7]: RangeIndex(start=0, stop=5, step=1)

You can access the associated values via the [ ] square brackets just like numpy however pandas.Series is much more flexible than the numpy counterpart that it emulates. They say imitation is the highest form of flattery.

Lets go grab some data from the LazyWebTM:

If one really thinks about the aspects of pandas.Series it is really a specialized version of a python dictionary. For those unfamiliar a dictionary (dict) is python structure that maps arbirtrary keys to a set of arbitrary values. Super powerful for data manipulation and data wrangling. Taking this is a step further pandas.Series is a structure that maps typed keys to a set of typed values. The typing is very important whereas the type-specific compiled code within numpy arrays makes it much more efficient than a python list. In the same vein pandas.Series is much more efficient python dictionaries. pandas.Series has an insane amount of commands:

Find Series Reference Here.

Next, we move to what i consider the most powerful aspect of pandas the DataFrame. A DataFrame is a data structure that organizes data into a 2-dimensional table of rows and columns, much like a spreadsheet. DataFrames are one of the most common data structures used in modern data analytics because they are a flexible and intuitive way of storing and working with data.

# Python code demonstrate creating
# DataFrame from dict narray / lists

import pandas as pd

# intialise data of lists.
data = {'Name':['Bob', 'Carol', 'Alice', ''],
'Age':[18, 20, 22, 24]}

# Create DataFrame
df = pd.DataFrame(data)

# Print the output.
print(df)
 [8]:
Name  Age
0    Bob   18
1  Carol   20
2  Alice   22
3          24       

Lets grab some data. nba.csv is a flat file of NBA statistics of players:

Get the NBA data file here.

i don’t watch or follow sports so i don’t know what is in this file. Just did a google search for csv statistics and this file came up.

# importing pandas package
import pandas as pd

# making data frame from csv file

# retrieving row by loc method
second = data.loc["R.J. Hunter"]

print(first, "\n\n\n", second)
[9]:
Team        Boston Celtics
Number                 0.0
Position                PG
Age                   25.0
Height                 6-2
Weight               180.0
College              Texas
Salary           7730337.0

Team        Boston Celtics
Number                28.0
Position                SG
Age                   22.0
Height                 6-5
Weight               185.0
College      Georgia State
Salary           1148640.0
Name: R.J. Hunter, dtype: object

How nice is this? Easy Peasy. It seems almost too easy.

For reference here is the pandas.Dataframe reference documentation.

Just to show how far reaching pandas is now in the data science world for all of you who think you may need to use Spark there is a package called PySpark. In PySpark A DataFrame is equivalent to a relational table in Spark SQL, and can be created using various functions. Once created, it can be manipulated using the various domain-specific-language (DSL) functions  much like your beloved SQL.

Which might be another Snake_Byte in the future.

i also found pandas being used in ye ole #HealthIT #FHIR for as we started this off csv manipulation. Think of this Snake_Byte as an Ouroboros.

This github repo converts csv2fhir ( can haz interoperability? ):

with pd.read_csv(file_path, **csv_reader_params) as buffer:
for chunk in buffer:

# increment the source row number for the next chunk/buffer processed
starting_row_num = chunk["rowNum"].max() + 1

chunk: Series = chunk.apply(_convert_row_to_fhir, axis=1)

for processing_exception, group_by_key, fhir_resources in chunk:
yield processing_exception, group_by_key, fhir_resources


So this brings us to the end of this Snake_Byte. Hope this gave you a little taste of a great python library that is used throughout the industry.

Muzak To Blog By:

Mike Patton & The Metropole Orchestra – Mondo Cane – June 12th 2008 (Full Show) <- A true genius at work!

One other mention on the Muzak To Blog By must go to the fantastic Greek Composer, Evángelos Odysséas Papathanassíou (aka Vangelis) who recently passed away. We must not let the music be lost like tears in the rain, Vangelis’ music will live forever. Rest In Power, Maestro Vangelis. i have spent many countless hours listening to your muzak and now the sheep are truly dreaming. Listen here -> Memories Of Green.