Jekyll2022-02-10T19:40:12+00:00https://mdswanson.com/atom.xmlMatt Swanson’s Blogblog, talks, book writeupsMatt Swansonswan3788@gmail.comAll software advice is wrong2020-10-31T00:00:00+00:002020-10-31T00:00:00+00:00https://mdswanson.com/blog/2020/10/31/every-article-about-software-is-wrong<h2 id="advice-without-context">Advice without context</h2>
<p>All software advice is wrong. The reason is <strong>context</strong> – or rather the lack of context.</p>
<p>Blog posts, articles, YouTube videos, Twitter threads, and conference talks rarely address the context in which the author is operating. Nuance is difficult to communicate and makes for less punchy writing (while also alienating audiences who realize “oh, that actually doesn’t apply to me…”).</p>
<p>Take the perennial topic:</p>
<blockquote>
<p>Which JavaScript framework should I use?</p>
</blockquote>
<p>Well, it depends.</p>
<p>Are you learning to code? Are you trying to get a job? Are you trying to get a job at a FAANG company? Are you working at a consultancy? A startup? Are you building a video game? Or a line of business application? Or one microservice in an enterprise CRM platform?</p>
<p>Are you a tech lead trying to balance the needs of the business with your ability to hire and train the three new grads that are starting on your project (which is already 2 months behind)?</p>
<p>Do you work for a Fortune 500 company doing an a digital transformation across 50 scrums teams made up of a rotating cycle of outsourced developer resources?</p>
<p>Are you picking a JavaScript framework to write a sample app for the Gumroad course you’re working on so that you can quit your job fixing legacy Wordpress sites?</p>
<p>Context matters more than any technical benchmark or design decision. But tweet-sized aphorisms are what sells.</p>
<p>Predict which article gets more eyeballs: “We’re abandoning React Native” vs “The many trade-offs we had to consider and ultimately what worked and what didn’t work for my team of six .NET developers building two native mobile apps for non-technical customers”</p>
<hr />
<h2 id="best-practices">Best practices</h2>
<p>Best practices become “best practices” because they achieve a good result across a wide range of contexts.</p>
<p>They should really be called <a href="https://en.wikipedia.org/wiki/Pareto_principle">Pareto Practices</a> (80/20 rule), but then we would have to acknowledge there are situations where continuous deployment, TDD, code reviews, agile, etc are not a silver bullet (and maybe even harmful) and that’s no fun.</p>
<p>The great part about best practices is that they give you shared language to talk about concepts quickly, but the worst part is that it’s easy to substitute an understanding of the first principles or assumed preconditions with the pre-packaged idea.</p>
<p>And how do things become accepted as best practices anyways?</p>
<ul>
<li>
<p><strong>It solves a problem</strong>: Common problems exist in software and knowing when to use a tool is an important skill. But do <em>you</em> have <em>that</em> problem? You sure?</p>
</li>
<li>
<p><strong>Community mindshare</strong>: It feels good to have other people share your excitment and to be part of a thriving ecosystem. Just remember that large companies have teams of “developer advocates” who are paid solely to promote, speak, and tweet about things the company has a vested interest in.</p>
</li>
<li>
<p><strong>That’s the old way</strong>: Some old code, libraries, practices, and tools are really terrible. I wouldn’t want to be writing COBOL. But not everything old is bad and not everything new (see also: “modern”, “lightweight”, “simple”) is good. Use <a href="https://en.wikipedia.org/wiki/Lindy_effect">time as a filter</a>; the good stuff will stand the test of time (and <a href="https://en.wikipedia.org/wiki/Antifragility">get better</a>).</p>
</li>
</ul>
<hr />
<h2 id="survivorship-bias">Survivorship bias</h2>
<p>You only see the extremes on the internet. And even then, it skews towards content that claims to, finally, have found the grand unified theory of building software.</p>
<p>It’s not a unique problem to programming. Instagram is a heavily edited, lopsided view of someone’s life. There is an ongoing <a href="https://www.sciencemag.org/news/2019/05/solution-psychology-s-reproducibility-problem-just-failed-its-first-test">replication crisis</a> in science because “we did an experiment and found nothing” isn’t a compelling preprint. For every business podcast about “how I made it”, 99 other entrepreneurs crashed and burned.</p>
<p>Once you’ve been around the block in this industry, you can remember the hype cycles that you lived through.</p>
<p>Remember NoSQL? It wasn’t all a meme. Redis is something good that emerged. People still use MongoDB sometimes. But what about Riak (ring-based clusters)? Neo4J (graph databases)? It wasn’t clear during the mania what was good and what was bad. Be careful when people take credit for being early adopters when there is also a unseen trail of abandoned codebases in the wake.</p>
<p>When was the last time you read an article from someone that explains how they really fucked up and got fired for making a bad decision? Super rare. And even if you do see someone repenting, you read till the end to find out that it’s all <a href="https://en.wikipedia.org/wiki/Kayfabe">kayfabe</a> to show how they saw the error of their ways, reformed, and <em>now</em> finally found The Right Tool For The Job.</p>
<hr />
<p>All software advice is wrong. I’ll check the comments to see why this one is too.</p>Matt Swansonswan3788@gmail.comAdvice without contextTwenty-five days of design2019-01-07T00:00:00+00:002019-01-07T00:00:00+00:00https://mdswanson.com/blog/2019/01/07/twenty-five-days-of-design<p>I spent the month of December completing the <a href="http://www.dailyui.co/">#DailyUI</a> challenges – a series of design prompts.</p>
<h2 id="why">Why?</h2>
<p>The idea of building things soup-to-nuts is very appealing to me.</p>
<p>I like to work on my own projects and I admire folks that are self-sufficient.</p>
<p>Not having to rely on others means that I have total control.</p>
<p>I think I have good design taste, but I wanted to push myself to “prove it”.</p>
<h2 id="selected-samples">Selected samples</h2>
<p>I finished 25 design challenges over the month, all designed in <a href="https://www.figma.com">Figma</a>.</p>
<p><img src="https://i.imgur.com/XeuOAqq.png" alt="" /></p>
<p>Here are a few designs that I think turned out well:</p>
<div class="caption">Day 03: Landing page</div>
<p><img src="https://mdswanson.com/static/daily-ui-1.png" alt="" /></p>
<div class="caption">Day 04: Forecasting calculator</div>
<p><img src="https://mdswanson.com/static/daily-ui-2.png" alt="" /></p>
<div class="caption">Day 13: Live chat debugging tool</div>
<p><img src="https://mdswanson.com/static/daily-ui-3.png" alt="" /></p>
<div class="caption">Day 16: Modal dialog</div>
<p><img src="https://mdswanson.com/static/daily-ui-4.png" alt="" /></p>
<div class="caption">Day 18: Analytics dashboard</div>
<p><img src="https://mdswanson.com/static/daily-ui-5.png" alt="" /></p>
<div class="caption">Day 22: Search autocomplete</div>
<p><img src="https://mdswanson.com/static/daily-ui-6.png" alt="" /></p>
<div class="caption">Day 23: Onboarding documentation</div>
<p><img src="https://mdswanson.com/static/daily-ui-7.png" alt="" /></p>
<hr />
<p>You can find all of my designs on this <a href="https://twitter.com/_swanson/status/1069311161391288321">Twitter thread</a> or <a href="https://dribbble.com/swanson">Dribbble</a>.</p>Matt Swansonswan3788@gmail.comI spent the month of December completing the #DailyUI challenges – a series of design prompts.Creative Selection: How Apple builds products with demos, taste, and leverage2018-11-26T00:00:00+00:002018-11-26T00:00:00+00:00https://mdswanson.com/blog/2018/11/26/creative-selection<p><a href="https://www.amazon.com/dp/B079DVT6VP">Creative Selection (“Inside Apple’s Design Process During the Golden Age of Steve Jobs”)</a> is a first-hand account of Ken Kocienda’s project history at Apple. Kocienda – a software engineer – describes his experiences building Safari, WebKit, and the first touch keyboard for the original iPhone and finds the commonalities in process that lead to success.</p>
<p>Don’t be mislead by the mention of “design process” in the book’s subtitle. This is not about <a href="https://www.cultofmac.com/188753/the-braun-products-that-inspired-apples-iconic-designs-gallery/">inspirational German appliances</a> or the <a href="https://www.theverge.com/2016/9/7/12838024/apple-iphone-7-plus-headphone-jack-removal-courage">‘courage’</a> to remove a headphone jack. The book is about the <em>product development cycle</em> at Apple, told from a technical perspective.</p>
<h1 id="key-takeaways">Key Takeaways</h1>
<h2 id="demo-driven-development">Demo driven development</h2>
<p>It is clear that, even across teams and products, demos trumped everything when it came to product decisions at Apple. There was a hierarchy of decision makers and you had to satisfy each one to move up the ladder. First, you demo to other engineers on your team, get feedback, and iterate. Next, you demo to a product manager, get feedback, and iterate. Only then would you get an opportunity to demo to the final authority: Steve Jobs.</p>
<p>There was a heavy emphasis on dogfooding the product and prototyping in code. Kocienda describes an entire day of ‘derby competitions’ to quickly prototype tons of competing options.</p>
<h2 id="taste-over-data-driven">Taste over data-driven</h2>
<p>To be successful at Apple, you needed taste. Kocienda defines having taste as “developing a refined sense of judgment and finding the balance that produces a pleasing and integrated whole”.</p>
<p>Taste can also be thought of as having the self-confidence to form opinions with your gut than you can justify with your head. You can build that confidence by studying great work. You have to <a href="https://soundcloud.com/seekingwisdom/sw-classics-carry-the-water">carry the water</a> to earn the right to have an opinion.</p>
<p>Contrast this approach with Google, who would A/B test twenty shades of blue to find the best converting hue. They have optimized for finding the local maximum – but that doesn’t lead to a “pleasing and integrated whole”.</p>
<p>As Kocienda puts it, “When it comes to choosing a color, we picked one. We used our good taste – and our knowledge of how to make software accessible… – and we moved on.” Commit, but be open to change.</p>
<h2 id="lessons-from-darwin">Lessons from Darwin</h2>
<p>The common principles across high-performing teams at Apple are an evolutionary ode to Charles Darwin.</p>
<ul>
<li>Start small, with some inspiration</li>
<li>Make demos</li>
<li>Mix in feedback</li>
<li>Seek guidance from smart colleagues</li>
<li>Blend in variations</li>
<li>Hone the vision</li>
<li>Follow the first demo with more and more demos</li>
<li>Improve in incremental steps</li>
<li>Converge on better and better versions</li>
</ul>
<h2 id="the-10x-engineer">The 10x Engineer</h2>
<p>This concept has been turned into a mix of a <a href="https://thenewstack.io/10x-programmer-just-jerk/">trope</a> and a <a href="https://twitter.com/search?q=10x%20engineer&src=typd">meme</a> by the software community, but Kocienda shares an ancedote about an engineer who made more progress on a tricky problem in his first two days than the rest of the team had made in six weeks. It wasn’t that this mythical engineer was ten times more productive, but rather that he was extremely good at evaluating many options quickly and picking the one that seemed most likely to bear fruit. This judgement, combined with ruthlessly cutting scope and taking shortcuts, allowed him to produce a demo that was convincing enough to explore an idea and move the product forward.</p>
<p>It still took many more months for the full implementation to get finished. And it was a slog. But the approach of getting something demoable and working crushed have nothing to show until the final reveal. The first demo created huge <strong>leverage</strong>: with a small amount of work, the results got executives on board, budgets allocated, and gave the team a baseline to iterate on.</p>
<h1 id="implications-at-a-software-consultancy">Implications at a software consultancy</h1>
<p>As with all books, one should map the lessons and experiences of the author into their own context.</p>
<p>I work at a software consultancy. There is a tension in consulting to produce value quickly. We want the “time to value” to be faster than our billing cycle. If the client is paying invoices but not seeing any value, that’s bad.</p>
<p>A consulting timeline can be volatile. We don’t often have the luxury of spending months and months to go extremely deep for one feature. A project (or our involvement with a project) can end suddenly, so it is wise to be able to wrap up a piece of work quickly.</p>
<p>None of this is meant to discount the process at Apple. Rather, it is to highlight that trying to copy the process wholesale will not produce identical results. We need to be more aware of the timelines and scale.</p>
<p>Could we take the whole team and prototype new interface components for weeks, culminating with multiple rounds of iterations and a bracket to let the best option win? Probably not. But could we spend a couple of hours trying out two or three options? Probably.</p>
<p>We need to treat projects like the 10x engineer: hitting the ground running, getting some small piece of software working, and then iterating.</p>
<p>Recently, my team used a third-party service to add full-text search to a client project. It took two days of work to get a result that was pretty good. Previous projects have allocated weeks or months for similar functionality. We should strive for that leverage – firms that deliver value in days (not months) eat the lunch of their competitors.</p>
<p>The most direct and actionable takeaway was that we should demo in-progress work more often. At Apple, engineers test their own software and demo to their co-workers constantly. I’ve seen client projects where demos happened infrequently (a handful of times a year) and only once work was completed. Even the ‘standard’ two-week cadence is starting to feel too slow.</p>
<p>I’m now trying as hard as I can to demo work at every daily standup meeting, even if I have to demo code from my local development environment. If I can’t demo, I show a screenshot or animated GIF.</p>
<p>Show, don’t tell. I could talk about what I’m working on…or I could show where I’m at. Ruthlessly demoing allows for tighter feedback loops and makes knowledge work more concrete. The customer can visually see what we’re working on and the progress we’re making.</p>
<p>Demos, unlike status reports or charts, don’t lie. An engineer can give the same vague update every day for a week and slide by, but not having anything to show for three days in a row sticks out like a sore thumb.</p>
<p>The biggest difference between the process at product companies and the realities of consulting is the mindset: innovation vs solid construction.</p>
<p>At Apple, Kocienda was truly innovating. There had never been a capable touchscreen keyboard before. There was no open source library to drop in or blog tutorial to follow. There were no competitors to draw inspiration from. Very little of the work at consulting firms is true innovation (despite what all the proposals and service offering pages say).</p>
<p>Our work is more akin to solid construction. Our clients are businesses who value consistency, reliability, and no bad surprises. Nearly all of their technical problems have well-known solutions. We can delight them and find places for novel ideas, but at the end of the day, it’s all about delivery.</p>
<p>Frequent demos, developing taste, and using technical leverage are great tools for delivering working software in a predictable way.</p>Matt Swansonswan3788@gmail.comCreative Selection (“Inside Apple’s Design Process During the Golden Age of Steve Jobs”) is a first-hand account of Ken Kocienda’s project history at Apple. Kocienda – a software engineer – describes his experiences building Safari, WebKit, and the first touch keyboard for the original iPhone and finds the commonalities in process that lead to success.The trouble with timestamps2018-10-22T00:00:00+00:002018-10-22T00:00:00+00:00https://mdswanson.com/blog/2018/10/22/trouble-with-timestamps<p><em>This tweet was sent about 2 hours ago.</em></p>
<p><em>Your order shipped about a week ago.</em></p>
<p><em>This file was last modified about 4 years ago.</em></p>
<p>The pretty “time ago” format has become popular on the web.</p>
<p>And it’s good…until it isn’t.</p>
<p>There are certain types of data where the pretty format is fine. It originally came from social networks, where the content was time-sensitive. When people talk about something posted on Facebook, they usually say something like “Did you see that picture Bill put on Facebook the other day?”. You would get funny looks if you said “Did you see that picture Bill posted on 2018-10-21T01:46:06+00:00?”</p>
<p>The timestamps matched the real-world format.</p>
<p>But what about <code class="language-plaintext highlighter-rouge">git</code> commits? What if I needed to see exactly when a change was introduced into my code? The bug report was entered on September 04, have we fixed it already? We deployed a build on September 01 at 2:30pm, what other changes also went out?</p>
<p><img src="http://i.imgur.com/Iz7yIMC.png" alt=""GitHub time ago"" /></p>
<p>A month ago doesn’t really tell me a whole lot in this case.</p>
<p><img src="http://i.imgur.com/HqkrfWq.png" alt=""GitHub full timestamp on hover"" /></p>
<p>(ProTip™: you can hover over dates on GitHub to see the full timestamp)</p>
<hr />
<p>Think about the context — most people don’t refer to status updates based on specific times or dates. But you’ll want to know the exact date and time when your bank sent a bill payment if you are on the phone with a collection agency.</p>
<p>What are the common use-cases for timestamped data in your app? Do users want a rough estimate or an exact value? Is the data short-lived or an important reference point?</p>
<p>The “time ago” style is prettier, but it isn’t always the most functional.</p>Matt Swansonswan3788@gmail.comThis tweet was sent about 2 hours ago.This committee is a pointless time suck2018-09-19T00:00:00+00:002018-09-19T00:00:00+00:00https://mdswanson.com/blog/2018/09/19/committee-pointless-time-suck<p>I’ve been on a dozen committees (or “company initiatives” or “recurring discussions” or “organizational development groups”) and their value has varied wildly.</p>
<p>In the past year, I’ve taken charge of one of the committees with the goal of making it run more effectively.</p>
<p>Here are some things I’ve been doing to keep my committees from becoming <strong>pointless time sucks</strong>.</p>
<h1 id="create-a-charter">Create a charter</h1>
<p>The biggest problem with committees is that they have no purpose. Not that they are <em>pointless</em>, but rather, the group does not clearly state <em>why</em> they formed and <em>what</em> they are trying to achieve.</p>
<p>Spend time upfront figuring out what you are trying to do and then put it in writing to confirm everyone is actually on the same page.</p>
<p><strong>If you can’t come up with a purpose statement and some guiding principles, you aren’t ready to have a committee.</strong></p>
<p>A charter is something to point back when you need to make a decision. Apply a test for any new ideas: does this align with the purpose of this group? Filter out things that don’t further the mission.</p>
<p>As an example, here is an excerpt from our Professional Development (PD) charter:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>The guiding principles that influence our PD system are:
* Minimizing friction to doing PD activities
* Minimizing friction to recording and sharing PD activities
* Setting the expectation that everyone does PD relevant to their current (or future) role
</code></pre></div></div>
<p>When trying to decide what activities the PD committee should support or create, they have to link to at least one of the principles. When we evaluate if something was a success, we judge it based on if it furthered the goals of the group, regardless of how fun or cool or exciting it was.</p>
<h1 id="no-agenda-no-meeting">No agenda? No meeting.</h1>
<p>Open-ended, recurring meetings are a great way to feel like you are getting things done. Everyone can walk in and chat about ideas and brainstorm, but it’s a trap.</p>
<p>Without an agenda, these meetings will never force a decision to be made. Especially when the people involved have other things on their plates, it’s easy to fall into a pattern of circling around the same topics for weeks.</p>
<p>The other problem with open-ended meetings is when everyone gets in the room and realizes there is nothing to talk about. The sunk cost fallacy sets in and <em>“since we’ve already got the time blocked off”</em>, we might as well just chat…</p>
<p>I’ve tried a few things to combat this and the one that worked best was a Slack reminder. The bot posts it a day before the meeting and <strong>if no one has any updates or items to discuss, we cancel the meeting</strong>.</p>
<p><img src="/static/slackbot-reminder.png" alt="" />
<em>An “are we meeting and what are we talking about?” reminder for a recurring twice-a-month meeting</em></p>
<hr />
<p><img src="/static/slackbot-reminder-2.png" alt="" />
<em>A “please come prepared or it will be a waste of everyone’s time” reminder</em></p>
<h1 id="build-a-road-map">Build a road map</h1>
<p>Fatigue is a challenge for committees. You can run a meeting on energy and enthusiasm for several months. But without <strong>being intentional and scoping new efforts</strong>, the excitement will fade and the group will slowly slide to a halt.</p>
<p>Something I’ve been trying recently and having success with is doing a formal quarterly road map. At the start of each quarter, we meet with the full group (scheduled off the regular cadence if necessary) to do three main items:</p>
<ul>
<li>Evaluate what activities we’re currently doing: are they effective? are they making an impact? are they still aligned to the charter?</li>
<li>Elevator pitches for new ideas or things to explore</li>
<li>Commit to which activities we are going to keep doing, which we are going to try for the quarter, and assign an “owner”</li>
</ul>
<p>For our group, the “owner” of an item was the one accountable for making sure it happened during the quarter, but not the sole responsible party. Everyone in the group is happy to help out, but having a designated person in charge makes sure it gets done.</p>
<p>These road maps are formal in the sense that we wrote it down, but still pretty lightweight (no metrics/OKRs/etc). Here is a road map from a past quarter:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Q2 Road Map:
* Keep doing current activities
** Quarterly PD sharing / PD site
** Processing reimbursement (ad-hoc)
** Upkeep on Commons display
** 2-week committee meetings
* Wrap up small items
** Retire #pd slack channel - MDS
** Write April newsletter - MCT
** Write May newsletter - TET
** Evaluate LMS - MCT
* New efforts
** Celebration Event - JCF
** Spring Cleaning 30 Day Challenge - MDS
** Uncover good PD stories - CRK
</code></pre></div></div>
<h1 id="cycle-members">Cycle members</h1>
<p>Some committees have a fixed lifespan – they form, do something, and then go away. Others are intended to be longer-term and continually support some activity or system.</p>
<p>For long-term groups, provide an “on/off ramp” for members. The group should ideally be cycling folks in and out over time to get a mix of different opinions and allow people to take a break or wind down as their own interests and availability wane.</p>
<p><strong>Making it out okay for people to take a break</strong> helps prevent burnout and people sticking around because they still want to support the effort or feel obligated to see it through till the end, even if they have lost their own excitement about the mission.</p>
<p>With a clear charter, it should be easier to change the group composition without losing alignment to the purpose.</p>Matt Swansonswan3788@gmail.comI’ve been on a dozen committees (or “company initiatives” or “recurring discussions” or “organizational development groups”) and their value has varied wildly.:hammer: Building a :bug: Fuzzy :smile: Emoji :mag: Finder :100:2018-07-06T00:00:00+00:002018-07-06T00:00:00+00:00https://mdswanson.com/blog/2018/07/06/fuzzy-emoji-finding<p>A while back I was on a project and, for whatever reason, our team started checking every morning to see what holiday it was and to bond over the absurdity of some of the holidays.</p>
<p>There are so many holidays, most of them are dumb or head-scratchers. Did you know that on the day I write this post it is (allegedly) <a href="https://www.youtube.com/watch?v=fZuJGhFhSV8">“Take Your Webmaster to Lunch Day”</a>? What did you do to celebrate everyone’s favorite special occasion <a href="https://foodimentary.com/2018/07/05/july-5th-is-national-apple-turnover-day-2/">“National Apple Turnover Day”</a>?</p>
<p>I decided that I needed to make a Slack bot to post the daily dumb holiday roundup. And this Slack bot should pick a relevant emoji for each holiday because emojis are fun.</p>
<p>As I went about over-engineering this dumb piece of useless software, it was surprisingly enjoyable and I was actually practicing relevant data science and NLP techniques.</p>
<p>So the goal: once a day, post a message to Slack with list of today’s holidays (and matching emojis) :tada:</p>
<h1 id="sourcing">Sourcing</h1>
<p>A Google search for <code class="language-plaintext highlighter-rouge">daily holidays</code> lead me to <a href="https://www.checkiday.com/">Checkiday.com</a> – a leading source of flimsy sourced holidays that I’m pretty sure are mostly made up.</p>
<p>I first thought about writing a scraper, but I found that our good friends over at Checkiday.com publish an RSS feed with the daily holidays: <a href="https://www.checkiday.com/rss.php?tz=America/Indianapolis">https://www.checkiday.com/rss.php?tz=America/Indianapolis</a></p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="n">feed</span> <span class="o">=</span> <span class="no">Excon</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="no">HOLIDAY_FEED</span><span class="p">)</span>
<span class="n">xml</span> <span class="o">=</span> <span class="no">Hash</span><span class="p">.</span><span class="nf">from_xml</span><span class="p">(</span><span class="n">feed</span><span class="p">.</span><span class="nf">body</span><span class="p">)</span>
<span class="n">source</span> <span class="o">=</span> <span class="n">xml</span><span class="p">[</span><span class="s2">"rss"</span><span class="p">][</span><span class="s2">"channel"</span><span class="p">][</span><span class="s2">"item"</span><span class="p">].</span><span class="nf">map</span><span class="p">{</span><span class="o">|</span><span class="n">i</span><span class="o">|</span> <span class="n">i</span><span class="p">[</span><span class="s2">"title"</span><span class="p">]}</span>
<span class="p">[</span>
<span class="s2">"Comic Sans Day "</span><span class="p">,</span>
<span class="s2">"International Kissing Day"</span><span class="p">,</span>
<span class="s2">"National Air Traffic Control Day"</span><span class="p">,</span>
<span class="s2">"National Fried Chicken Day®"</span><span class="p">,</span>
<span class="s2">"Take Your Webmaster to Lunch Day"</span><span class="p">,</span>
<span class="s2">"Umbrella Cover Day"</span>
<span class="p">]</span>
</code></pre></div></div>
<h1 id="cleaning--normalization">Cleaning / normalization</h1>
<p>The first step in my data pipeline is to <a href="https://en.wikipedia.org/wiki/Text_normalization">clean and normalize the input data</a>. In this case, that means pulling the holiday names from the RSS feed and then striping extra whitespace, removing special characters, and making everything lowercase.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="n">source</span><span class="p">.</span><span class="nf">map</span> <span class="k">do</span> <span class="o">|</span><span class="n">holiday</span><span class="o">|</span>
<span class="n">holiday</span>
<span class="p">.</span><span class="nf">downcase</span>
<span class="p">.</span><span class="nf">strip</span>
<span class="p">.</span><span class="nf">gsub</span><span class="p">(</span><span class="sr">/[^a-z ]/i</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
<span class="k">end</span>
<span class="p">[</span>
<span class="s2">"comic sans day"</span><span class="p">,</span>
<span class="s2">"international kissing day"</span><span class="p">,</span>
<span class="s2">"national air traffic control day"</span><span class="p">,</span>
<span class="s2">"national fried chicken day"</span><span class="p">,</span>
<span class="s2">"take your webmaster to lunch day"</span><span class="p">,</span>
<span class="s2">"umbrella cover day"</span>
<span class="p">]</span>
</code></pre></div></div>
<h1 id="stop-words">Stop words</h1>
<p><a href="https://en.wikipedia.org/wiki/Stop_words">Stop words</a> are common words that don’t provide any semantic value to a chunk of text (words like “the”, “and”, “of”, etc). These should be removed from the input to remove noise.</p>
<p>I found a handy Ruby gem with a bunch of stopwords: <a href="https://github.com/brez/stopwords">https://github.com/brez/stopwords</a> so I added the gem and removed individual stop words from the holiday titles.</p>
<p>Another common operation is to remove domain specific stop words. If you were looking at medical data, you might remove common low-value words like “doctor”, “dr”, “patient”, etc. In my case, I looked at a few days worth of holidays and made a custom stop list with words like “national”, “day”, “international”, “festival”, etc.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="n">source</span><span class="p">.</span><span class="nf">map</span> <span class="k">do</span> <span class="o">|</span><span class="n">holiday</span><span class="o">|</span>
<span class="n">holiday</span>
<span class="p">.</span><span class="nf">downcase</span>
<span class="p">.</span><span class="nf">strip</span>
<span class="p">.</span><span class="nf">gsub</span><span class="p">(</span><span class="sr">/[^a-z ]/i</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
<span class="p">.</span><span class="nf">split</span><span class="p">(</span><span class="s2">" "</span><span class="p">)</span>
<span class="p">.</span><span class="nf">reject</span><span class="p">{</span><span class="o">|</span><span class="n">w</span><span class="o">|</span> <span class="no">Stopwords</span><span class="p">.</span><span class="nf">is?</span> <span class="n">w</span> <span class="p">}</span>
<span class="p">.</span><span class="nf">reject</span><span class="p">{</span><span class="o">|</span><span class="n">w</span><span class="o">|</span> <span class="no">HolidayStopwords</span><span class="p">.</span><span class="nf">is?</span> <span class="n">w</span> <span class="p">}</span>
<span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="s2">" "</span><span class="p">))</span>
<span class="k">end</span>
<span class="p">[</span>
<span class="s2">"comic sans"</span><span class="p">,</span>
<span class="s2">"kissing"</span><span class="p">,</span>
<span class="s2">"air traffic control"</span><span class="p">,</span>
<span class="s2">"fried chicken"</span><span class="p">,</span>
<span class="s2">"webmaster lunch"</span><span class="p">,</span>
<span class="s2">"umbrella cover"</span>
<span class="p">]</span>
</code></pre></div></div>
<h1 id="n-grams">N-grams</h1>
<p>Now I have a list of keywords for the given holidays that has been stripped of most of the noise. After some trial and error, I found that doing n-grams helped improve the pipeline results.</p>
<p><a href="https://en.wikipedia.org/wiki/N-gram">N-grams</a> are a fancy way of saying “groups of words”. It is a contiguous sequence of words in your text. You can generate bigrams (each 2 word pair) or trigrams (each 3 word pair) or even more if you want.</p>
<p>There is probably some gem for this but I didn’t even really need full n-grams, so what I did was just split the holidays by spaces to generate some kinda-sorta-n-grams. It’s close enough.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="k">def</span> <span class="nf">ngram</span><span class="p">(</span><span class="n">phrase</span><span class="p">)</span>
<span class="n">phrase</span><span class="p">.</span><span class="nf">split</span><span class="p">(</span><span class="s2">" "</span><span class="p">).</span><span class="nf">append</span><span class="p">(</span><span class="n">phrase</span><span class="p">)</span>
<span class="k">end</span>
<span class="s2">"air traffic control"</span> <span class="o">-></span> <span class="p">[</span><span class="s2">"air"</span><span class="p">,</span> <span class="s2">"traffic"</span><span class="p">,</span> <span class="s2">"control"</span><span class="p">,</span> <span class="s2">"air traffic control"</span><span class="p">]</span>
</code></pre></div></div>
<h1 id="semantic-keywords">Semantic keywords</h1>
<p>There are tons of emojis, but there are even more English words so not every holiday was going to find an exact (or even close) match.</p>
<p>Take the :hamburger: emoji for instance. This is the right emoji to use for “hamburger”, but also “cheeseburger”, “burger”, “fast food”, etc</p>
<p>If I tried to match “National Cheeseburger Awareness Day” it would probably find something like :cheese: instead of the :hamburger:, because computers are really dumb.</p>
<p>Luckily, I found the Emojilib project: <a href="https://github.com/muan/emojilib">https://github.com/muan/emojilib</a> where the internet has lovingly added keywords to over 5000+ emojis.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w">
</span><span class="nl">"hamburger"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"keywords"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"meat"</span><span class="p">,</span><span class="w"> </span><span class="s2">"fast food"</span><span class="p">,</span><span class="w"> </span><span class="s2">"beef"</span><span class="p">,</span><span class="w"> </span><span class="s2">"cheeseburger"</span><span class="p">,</span><span class="w"> </span><span class="s2">"mcdonalds"</span><span class="p">,</span><span class="w"> </span><span class="s2">"burger king"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"char"</span><span class="p">:</span><span class="w"> </span><span class="s2">"🍔"</span><span class="p">,</span><span class="w">
</span><span class="nl">"fitzpatrick_scale"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"category"</span><span class="p">:</span><span class="w"> </span><span class="s2">"food_and_drink"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>I downloaded the full listing of emojis and create a mapping of keywords to emojis. Real professionals might call this a <a href="https://en.wikipedia.org/wiki/Semantic_search">semantic mapping</a>, but I’m not a real professional.</p>
<h1 id="fuzzy-match-scoring">Fuzzy match scoring</h1>
<p>Now that I’ve pruned the holidays down to the most relevant parts and I have a huge dictionary of keywords (and their corresponding emojis), it’s time to find the closest match for each holiday.</p>
<p>I grabbed the <code class="language-plaintext highlighter-rouge">fuzzy_match</code> gem: <a href="https://github.com/seamusabshere/fuzzy_match">https://github.com/seamusabshere/fuzzy_match</a> which takes a “needle” and a “haystack” and then uses a few different algorithms to return the closest matching emoji keyword.</p>
<p>I ran the fuzzy matcher for each of our n-grams from above and picked the keyword with the highest matching score. If multiple items had exact matches (e.g. “chocolate” and “milk”), I picked the match that was the longest word. Why? I don’t know, it seemed good.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
Input: make gingerbread house
=> [["cake", 0.66],
["bread", 0.57],
["house", 1.0],
["derelict_house_building", 0.41]]
</code></pre></div></div>
<h1 id="mechanisms">Mechanisms</h1>
<p>I glued together a script to pull the holidays from the RSS link, feed them into my very fancy :tm: data pipeline, and then format and post the message to <a href="https://api.slack.com/bot-users">Slack via a bot user</a>. I put the app on Heroku and used the <a href="https://elements.heroku.com/addons/scheduler">Scheduler</a> to run the script once a day.</p>
<h1 id="results">Results</h1>
<p>For ~75 lines of code, the results are really, really good. The emoji suggestions are sometimes even spooky good.</p>
<p>Here is a sampling of the test data:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
:bear: Polar Bear Plunge Day
:stuffed_flatbread: Buffet Day
:eyes: Run It up the Flagpole and See If Anyone Salutes It Day
:microscope: Science Fiction Day
:wastebasket: Fruitcake Toss Day
:new_moon: Festival of Sleep Day
:bird: Bird Day
:globe_with_meridians: Earths Rotation Day
:shower: Clean Off Your Desk Day
:bulb: Static Electricity Day
:battery: Cut Your Energy Costs Day
:zzz: Make Your Dreams Come True Day
:house: Organize Your Home Day
:strawberry: Strawberry Ice Cream Day
:face_with_no_mouth: Nothing Day
:link: Benjamin Franklin Day
:id: Kid Inventors Day
:new: Ditch New Years Resolution Day
:taurus: Thesaurus Day
</code></pre></div></div>
<p>And the finished bot:</p>
<p><img src="https://i.imgur.com/2daeAga.png" alt="" /></p>
<p><img src="https://i.imgur.com/xhHFa0x.png" alt="" /></p>
<h1 id="conclusion">Conclusion</h1>
<p>This was a very silly project, but it was really fun and I think it is illustrative of the type of work you can do to get a pretty good result with minimal effort.</p>
<p>You can see the code here: <a href="https://github.com/swanson/dumb-holidays">https://github.com/swanson/dumb-holidays</a></p>
<p>Possible improvements and further over-engineering: stemming, better n-grams, ensemble fuzzy matching algorithms, better keyword corpus mapping, sentiment analysis, blockchain, :poop:</p>Matt Swansonswan3788@gmail.comA while back I was on a project and, for whatever reason, our team started checking every morning to see what holiday it was and to bond over the absurdity of some of the holidays.Evaluating project opportunities2018-05-16T00:00:00+00:002018-05-16T00:00:00+00:00https://mdswanson.com/blog/2018/05/16/evaluating-project-opportunities<p>I’ve been having conversations at <a href="https://www.sep.com/">work</a> about how to evaluate software project opportunities for our firm.</p>
<p>We can talk about things like what the project is, whether or not we think the client will be good to work with, or the size and budget of the company. These are all useful criteria when talking about opportunities during strategic sales discussions.</p>
<p>But something that I specifically have been asked to help with is how to determine if engineers would find the project enjoyable. Would it be fun to work on?</p>
<p>I think there are a few levels of thinking required to answer this question, each more refined and nuanced.</p>
<h2 id="level-0-no-opinion">Level 0: No opinion</h2>
<p>Early in your career you’re still figuring out how to build software, how to work on a team, and generally finding your footing. You might not have much preference about what you work on. In an optimistic case, you’re happy to work on anything! It’s all new and exciting. In a pessimistic case, you have no frame of reference for what an enjoyable and enriching project should feel like so you can’t identify red flags.</p>
<h2 id="level-1-surface-level-attributes">Level 1: Surface level attributes</h2>
<p>After a while you start to form opinions and preferences on the type of work you like. This manifests with statements like <em>“I love coding in Ruby”</em> or <em>“I don’t like embedded development”</em> or <em>“My ideal project would be building software for spaceships”</em>.</p>
<p>These kind of attributes are easy to spot when assessing an opportunity. Most briefings will include the domain, tech stack, or other details that allow an engineer to judge the project based on their own interests: this would “check the boxes” for me because it’s a .NET web app with Angular.</p>
<p>We can build a list of keywords that less technical folks can look for. We can kill opportunities that match some kind of ‘blacklist’ and elevate ones that perk up our ears.</p>
<h2 id="level-2-foundational-attributes">Level 2: Foundational attributes</h2>
<p>Eventually, you’ll have the opportunity to work on a project with good surface level attributes. Unfortunately, you might find that despite checking the boxes, the project was not fun or enjoyable.</p>
<p>You might work with an amazing spaceship startup and a cutting edge tech stack, but find that you are constantly blocked by external dependencies, expected to work overtime, and not given authority for important decisions that impact your team.</p>
<p>This an extreme. But I have found that when I look back on my own career, there were projects that looked awesome on paper and turned out to be not so great (and vice-versa). Deep down we know this to be true; often how we feel about a project has little to do with the specific technology we are working with.</p>
<p>The foundational attributes of project enjoyment are deeper and more difficult to evaluate. We can try applying mental models like <a href="https://en.wikipedia.org/wiki/Drive:_The_Surprising_Truth_About_What_Motivates_Us">Daniel Pink’s motivation trio of autonomy, mastery, and purpose</a>.</p>
<p>Autonomy</p>
<ul>
<li>Will we be able to make our own choices on tooling, techniques, and practices?</li>
<li>How much do we ‘own’ the success of the project? How much are we depending on things or people outside of our control to make progress?</li>
<li>How will work be planned, budgeted, and distributed among the team(s)? What influence do we have in the direction of the project?</li>
<li>Is the client bringing a “problem to be solved” or a “spec to be implemented”?</li>
</ul>
<p>Mastery</p>
<ul>
<li>Will we have the opportunity to gain expertise in a modern/desirable tech stack?</li>
<li>What new skills will we be able to practice? What kind of knowledge can we acquire?</li>
<li>What sort of leadership opportunities will be available for the team? Can this project support a person new to leadership getting some reps?</li>
<li>Do we have sufficient foundational knowledge to execute the project on a technical level?</li>
</ul>
<p>Purpose</p>
<ul>
<li>Who are the end users? Internal employees? Other developers? Customers? No one?</li>
<li>Can we understand and empathize with the domain? Is there sufficient interest to dive deep and live in the client’s world?</li>
<li>Are we aligned on the practices and values that matter?</li>
<li>Are we making this small corner of the software world a better place?</li>
</ul>
<hr />
<p>When assessing project opportunities we are working with incomplete information. It is certainly easy to weigh only the surface level attributes of a project; it’s easy to explain, it’s easy to filter, and it’s easy to pattern match.</p>
<p>But the trouble is that most of these attributes are superficial. We know that two projects with an identical tech stack or for the same client will have differences in how fulfilling and enjoyable they are. In some cases, the difference is huge!</p>
<p>It’s hard to go beneath the surface. Some of these attributes take years of experience to uncover; there is a lot of trial and error and introspection involved. And even once we’ve found a satisfactory list of questions, most of them are tricky to answer. They are less “<em>yes or no</em>” and more “<em>well, maybe, it depends</em>”.</p>
<p>I won’t assume that I’ve found the ideal model yet, but digging deeper into what actually makes for a “good” project is a worthwhile exercise. For my <a href="https://www.sep.com/">company</a>, our average project duration and sales pipeline means that our evaluations of project opportunities <strong><em>right now</em></strong> will impact the work we’ll be doing over the next 3-5 years. I’m hard-pressed to think of a higher leverage activity to be thinking about and trying to improve.</p>Matt Swansonswan3788@gmail.comI’ve been having conversations at work about how to evaluate software project opportunities for our firm.How to start a peer group2017-03-29T00:00:00+00:002017-03-29T00:00:00+00:00https://mdswanson.com/blog/2017/03/29/how-to-start-a-peer-group<p>A super tactical list for starting a <a href="http://blog.drift.com/peer-groups">peer group</a>.</p>
<hr />
<h2 id="organizing-the-group">Organizing the group</h2>
<ol>
<li>
<p>Pick two people (that you already know) that you want in the group – invite them</p>
<blockquote>
<p>Hey, I’m organizing a peer group focusing on <strong>topic</strong> – are you interested in participating in a small group discussion with other local folks?</p>
</blockquote>
</li>
<li>
<p>Ask those people to recommend two people they want to be a part of the group – invite them, too</p>
</li>
<li>
<p>You should have 6-7 people now – if not, keep asking people that are interested to recommend people to invite</p>
</li>
<li>
<p>Send out three dates for a half-day meeting with <a href="http://doodle.com/">Doodle</a> – pick the soonest date that everyone can make</p>
</li>
<li>
<p>Email everyone (in one email thread) with the date and meeting agenda – this is now your discussion forum</p>
</li>
<li>
<p>You will host the first meeting so book the space and handle food</p>
<h2 id="day-of-the-first-meeting">Day of the first meeting</h2>
</li>
<li>
<p>You will volunteer to be the moderator of the first meeting</p>
</li>
<li>
<p>Start on time and have everyone introduce themselves – name, company, role, what they want to get out of the group</p>
</li>
<li>
<p>Set the rules of engagement – use <a href="https://en.wikipedia.org/wiki/Chatham_House_Rule">Chatham House Rule</a> unless there is a strong reason not to</p>
</li>
<li>
<p>Explain the discussion format – use <a href="http://agilecoffee.com/leancoffee/">Lean Coffee</a> unless there is a strong reason not to</p>
</li>
<li>
<p>Have a productive, educational, energizing, and cathartic discussion with your new peer group</p>
</li>
<li>
<p>Get a volunteer to host next time (quarterly works well)</p>
</li>
<li>
<p>Email everyone thanking them for their participation and enjoy the rest of your day</p>
</li>
<li>
<p>Repeat quarterly, expanding the group with new people as necessary</p>
</li>
</ol>
<hr />
<p>If you’ve started a peer group with this list, ping me <a href="https://twitter.com/_swanson">on Twitter</a> and let me know how it went.</p>Matt Swansonswan3788@gmail.comA super tactical list for starting a peer group.Software ate the world2017-02-14T00:00:00+00:002017-02-14T00:00:00+00:00https://mdswanson.com/blog/2017/02/14/why-machine-learning-is-interesting<p>It’s been over five years since Marc Andreessen proclaimed that <a href="https://www.wsj.com/articles/SB10001424053111903480904576512250915629460">“software is eating the world”</a>. And he was right. Every person I talk to is using software in their job.</p>
<p>When software started eating the world, there was low-hanging fruit. All the paper was removed. Even recreational sports leagues stopped being managed via phone calls and legal paper and started moving to the cloud.</p>
<p>A touted approach for finding software product ideas was taking an Excel sheet that was emailed back and forth and turning it into a SaaS product.</p>
<p>And this progress was, for the most part, great.</p>
<p>But it’s been five years.</p>
<p>The stuff that was easy to turn into software has been, well, turned into software. What remains is the hard stuff. The hard stuff needs expert, tacit knowledge.</p>
<p><strong>While the current wave of software can make us more efficient, it struggles to make us more effective.</strong></p>
<p>Fifteen years ago, a manager was delighted to have electronic tool to track project status. And five years ago, they were delighted by a beautiful interface and powerful reports they could create. But now, to delight them we need software that anticipates a scheduling bottleneck before the manager even thinks about it.</p>
<p>Beautiful, functional products are now table stakes. Customers want predictions, insights, and recommendations.</p>
<p>Software ate the world because it provided value and solved problems. If we want software to keep providing more value and solving more problems, we have to shift toward building things that bring expert knowledge into our products.</p>
<p>And for that, we need machine learning.</p>Matt Swansonswan3788@gmail.comIt’s been over five years since Marc Andreessen proclaimed that “software is eating the world”. And he was right. Every person I talk to is using software in their job.Algorithms to Live By2017-02-14T00:00:00+00:002017-02-14T00:00:00+00:00https://mdswanson.com/writeup/2017/02/14/algorithms-to-live-by<p><strong>What’s the point?</strong></p>
<p>“Algorithms to Live By” covers a wide breadth of computer science and machine learning topics (caching, halting problem, Big O, Bayesian statistics, NP-completeness, etc) – but it frames each topic as a tool for understanding or solving real-world problems. It’s basically <a href="http://mdswanson.com/writeup/2012/07/25/the-personal-mba.html">“The Personal MBA”</a> for computer science. The format is superb – enough detail to get readers familiar with a concept and be able to recognize potential uses, while providing source material references for when you need to go deeper.</p>
<p><strong>What are my takeaways?</strong></p>
<p>There are a lot of problems we encounter in our personal and work life that could be solved with basic algorithms. The optimal stopping algorithm can help us know when we’ve seen enough candidates for a job position. Caching algorithms can help you clean out old clothes in your closet. Sometimes just doing things randomly can you get very close to an optimal result, for a fraction of the effort.</p>
<p>I’m less interested in sorting my sock drawer with a merge-sort, but seeing examples of how to apply algorithms served as a useful guideline for finding non-silly applications.</p>
<p>The chapter on Bayes’ Rule was my favorite. My biggest insight was the discussion of how to consider the distribution of a system when trying to estimate. Even knowing if something is Gaussian, power-law, or Erlang distributed can have a huge impact in how we predict things.</p>
<p>Consider the following examples:</p>
<p>If a boy is currently 6 years old, how long should we predict they will live if we know nothing else about them? Since we know life expectancy is a roughly normal distribution, we should predict they will live to be a little more than 78 (the average, plus a bit more since they aren’t in the far-left tail).</p>
<p>Now what if we need to predict how much a movie will gross at the box office given it did $12M in its opening weekend? If we try to use the average like for predicting age, we will have a very high error because box office totals follow a power-law distribution. Instead, we should use a multiplicative factor.</p>
<p>I found the book to be really enjoyable and helped me breach topics that I had previously ignored because of a high barrier to entry. Coming from an engineering background (not a CS background), I really appreciate this style of teaching topics through practical examples.</p>Brian Christian and Tom GriffithsWhat’s the point?