Mike Durst

In this article, an exposed (passwordless) MongoDB tied to an ID verification flow allegedly left ~1B identity records visible across 26 countries—203M+ in the US alone. Not “just emails”: names, addresses, DOB, national IDs/SSNs, phone numbers… the exact ingredients for SIM-swaps and phishing that knows your home address. The wild part: it was secured the next day, but bots scan for this stuff and can copy it in minutes. Also: you probably never heard of the vendor—your bank/app did. That third-party chain is the blast radius. Are you still using SMS for 2FA anywhere? Why? #KYC #IdentityVerification #DataBreach #SIMSwap #MongoDB #FraudPrevention #ThirdPartyRisk #PrivacySecurity Security is a streak you can’t afford to break.

1

1 Comment

We are very excited to release Twig 0013 Cypress our latest RAG Model. With Document Priority, Reranking and separate flows for Retrieval and Responses. Release notes below. #GenAI #LLM #RAG #AIAgents We’re releasing Cypress, a new retrieval system built to give stronger, more consistent answers when your knowledge lives across many sources like Confluence, Help Centers, Slack, Teams, and long-form documents. Why we built Cypress As teams accumulate more content, a few issues tend to appear: High-value sources get buried under everyday chat noise Vague or unclear questions lead to irrelevant matches Large documents lose out to short messages in vector search Cypress is designed to address these challenges directly. Whats new is Cypress 1. Clearer understanding of the question, It rewrites ambiguous queries so retrieval focuses on the intended meaning. 2. Prioritized retrieval from your best sources, Official docs and primary knowledge bases are weighted above lower-value sources. 3. Broader candidate search, More potential matches are considered before ranking, improving overall coverage. 4. Smarter ranking, A new ranking model selects answers based on meaning and context—not just vector closeness. What improves - More accurate answers, especially for unclear questions - Your most important content is surfaced consistently - Broader retrieval without increasing noise - Cleaner, more reliable responses Use Cypress if: Answer quality matters more than speed You manage diverse source types Users phrase questions in different ways A small increase in latency (3–4s) is acceptable

19

1 Comment

Following on from my post on indexes and open table format performance, I realized that real-time analytics systems such as ClickHouse take slightly different approaches. Last year I did a deep dive into ClickHouse, including how it lays out data, its indexing structures, and data-skipping mechanisms. ClickHouse shares some philosophies like data layout and pruning with open table formats like Iceberg but goes further by adding sparse indexing and a tightly integrated storage engine rather than a headless storage format. Key characteristics: 🔹 Part-based storage: A table consists of Parts, self-contained data units. Each Part is sorted by primary (or sort) key and contains its own indexes, skip indexes, and optional projections. 🔹 Sparse indexing: The primary index stores one entry per granule (≈8K rows). Each entry maps a key range to a physical offset in the Part, enabling fast binary search. Secondary data-skipping structures (min/max, bloom filters, etc.) summarize statistics across multiple granules. These indexes are scoped to the Part, not global structures. 🔹 In-part projections: Aggregations and materialized projections can be maintained within Parts as separate files, avoiding external materialized views. 🔹 Continuous merging: Parts are immutable but continuously merged into larger Parts, preserving order, improving compression, and index efficiency. The index/pruning structures are most efficient in few larger Parts, than a multitude of small ones. 🔹 Part-scoped pruning: Query pruning is driven by each Part’s metadata (stored in memory). The system functions as a collection of self-contained units rather than a single global index. There are parallels with Iceberg: Parts resemble Iceberg data files with sidecar metadata, and merging resembles compaction. But they target different workloads. ClickHouse is optimized for low-latency, ad-hoc analytics and interactive dashboards, where fine-grained pruning and primary-key ordering matter. Iceberg is optimized for large-scale batch analytics and data sharing across engines, with coarser pruning and externalized metadata. Read my ClickHouse deep dive here: https://lnkd.in/d_Myhphf

50

4 Comments

📝 Blogged: "You Gotta Push If You Wanna Pull" Wrote up some thoughts on push vs. pull queries, materialized views, and why data duplication isn't something to fear. (Link to the post in the first comment below 👇)

46

2 Comments

The other day, I needed a new storage layer implementation. I decided to give GitHub's spec-kit a spin. The questions it asked about functional and non-functional requirements were thoughtful. The code it generated wasn't ready. I ran three iterations, each with more nuanced requirements. It got closer but still missed, so I ended up writing it by hand. The takeaway: 𝗮𝗴𝗲𝗻𝘁𝘀 𝗱𝗼 𝗯𝗲𝘁𝘁𝗲𝗿 𝘄𝗵𝗲𝗻 𝘆𝗼𝘂 𝗴𝗶𝘃𝗲 𝘁𝗵𝗲𝗺 𝗮 𝗿𝗲𝗮𝗹 𝗰𝗼𝗻𝘁𝗿𝗮𝗰𝘁. Interfaces, a defined schema, and a few tests help a lot, especially on anything that touches storage. Next time I'll start with a smaller scope, write the interface and data types up front, then let the agent fill in the middle. That way each iteration still teaches me something even if the code doesn't land. What has helped you get reliable results on backend tasks with coding agents? #DeveloperTools #SoftwareDevelopment #AI

11

3 Comments

After managing engineers for years, I discovered something ironic: Team leads are crushing it with AI while senior ICs struggle. Why? Managing people taught them exactly what LLM needs: Breaking projects into clear subtasks Explaining context without assumptions Giving specific, actionable feedback Iterating based on results The delegation skills you built for humans? They're your AI superpower. The best engineers might be prompt-engineering wrong because they never learned to delegate.

42

3 Comments

Semantic Vectors as a Reframing of SDLC What if issues in complex systems arise not only from implementation, but from the way problems are framed long before development begins? How do we structure our thinking so that we don’t narrow the solution space too early, and remain open to more coherent architectural paths? One possible approach is to explicitly articulate semantic levels of understanding before entering the development phase. 1. What are we actually trying to do? This is not yet architecture or design. Here we define a conceptual umbrella - the space within which further reasoning takes place. Example: A payment routing rule management system with specific business constraints. At this stage, we deliberately avoid solutions. We align on intent, boundaries, and why this problem exists at all. 2. How do we intend to approach it? Now we start connecting business intent with technical reality. This is where an architectural trajectory emerges - not as a diagram yet, but as a semantic path: - which constraints matter, - which trade-offs are acceptable, - which principles should guide decisions. The goal here is coherence, not completeness. 3. What does this affect? Once the trajectory becomes visible, we can zoom out: - Which parts of the system are impacted? - Where do new questions appear? - What assumptions need validation? This step often reveals hidden connections, organizational friction, or future risks - before any code is written.. These steps form an iterative semantic cycle: - refining initial conditions, - adjusting the trajectory, - minimizing unnecessary complexity, - reducing unintended side effects. Only when business intent and architectural understanding converge does it make sense to enter a standard SDLC flow. In practice, this semantic phase is often implicit. Making it explicit helps teams reason more clearly before committing to solutions. --- These principles are articulated as foundational semantic vectors within Harmogics, reflecting recurring patterns in how complex systems are reasoned about and designed. This article explores how such principles may be applied to rethinking the processes that shape software development before implementation begins.

2

Anthropic just shipped file-system agents to consumers. The architecture matters more than the feature! Cowork takes Claude Code's autonomous capabilities and removes the CLI barrier - now anyone on macOS can point Claude at a folder and let it execute multi-step workflows. Organize files, process data, create documents. The AI plans and executes independently. Why the technical approach matters Most AI tools operate in browser sandboxes. Cowork works at the OS level - executing file operations autonomously, not just suggesting edits. That's a fundamentally different execution model. Anthropic built this in ~1.5 weeks using Claude Code itself. When AI can scaffold its own tooling, development velocity changes completely. What this signals strategically The AI productivity market is bifurcating: 1. Chat-based tools scale horizontally (more users, more conversations, commoditizing) 2. Agentic tools scale vertically (fewer users, higher value, defensible if you solve for trust) Anthropic's $100-200/month pricing reflects this - they're positioning Cowork as workflow infrastructure, not consumer SaaS. The constraint is trust, not capability Claude can delete files and execute arbitrary operations. That's why this is a controlled research preview. The technical primitive works - the challenge is building trust infrastructure to scale it. Audit logs, rollback mechanisms, granular permissions - these are requirements, not nice-to-haves. Why OS-level access is the real trend Browser AI is limited by web security models. It can't access local files, email databases, or coordinate between applications without manual intervention. You feed it context, then manually move its output somewhere useful. OS-level AI sees your complete working context and executes full workflows autonomously. Pull data from email, update a spreadsheet, draft a report, sync to cloud - without human intervention between steps. Most valuable work happens between applications. Browser AI lives in one tab. System-level AI orchestrates across your entire computing environment. This is why the trend isn't just "better AI" - it's where AI runs. Browser tools compete on inference quality and UI (commoditizing fast). OS-level tools compete on integration depth and trust infrastructure (defensible, takes years to build). Where this leads If file-system agents prove out, next is inter-application orchestration. Today Cowork organizes folders. Tomorrow it coordinates across your file system, calendar, email and task manager. The highest-leverage AI applications won't live in browsers - they will live in your system with administrator-level permissions. Cowork isn't just a productivity feature. It's Anthropic testing whether users are ready for AI with OS-level access. https://lnkd.in/gAQG9UGe https://lnkd.in/gXW756p7 #AgenticAI #AI #ArtificialIntelligence

6

1 Comment

Mike Merrill and I are excited to announce the next chapter of Terminal-Bench with two releases: 1. Harbor, a new package for running sandboxed agent rollouts at scale 2. Terminal-Bench 2.0, a harder version of Terminal-Bench with increased verification Harbor is the package we wish we had had while making Terminal-Bench. It’s for agent, model, and benchmark developers and researchers who want to evaluate and improve agents and models. Just a few of the features I love about Harbor: - Evaluate any agent that can be installed and run autonomously - Scale up to thousands of concurrent containers using providers like Daytona and Modal - Generate rollouts for SFT data gen and RL - Create your own benchmarks or use existing ones like SWE-Bench Verified Harbor is also the official harness for Terminal-Bench 2.0. We used it to run tens of thousands of experiments in containerized environments while developing the latest version of the benchmark So why Terminal-Bench 2.0? We always knew that as model capabilities increased, we’d need to keep Terminal-Bench up to date with frontier capabilities. Terminal-Bench 2.0 consists of 89 hard tasks to test these capabilities. We aim to push the frontier with an increased emphasis on task quality. Each task received several hours of human and LM-assisted verification to ensure that tasks are (1) solvable, (2) realistic, and (3) well-specified. We’ll share more on how we did this in our upcoming preprint. At present, Codex CLI with GPT-5 sits at the top of our new leaderboard. Astute Terminal-Bench fans may notice that SOTA performance is comparable to TB1.0 despite our claim that TB2.0 is harder. We believe that this is because task quality is substantially higher in the new benchmark. We have removed several misspecified or impossible tasks – increasing difficulty while maintaining raw performance. Interested in using Terminal-Bench-2.0 or submitting to our new leaderboard? Check out the Harbor docs for more information. It's been a pleasure working alongside our advisors, Ludwig Schmidt, who seeded the idea and guided our development, and Andy Konwinski, who gave us the Laude Institute mandate to "ship your research." Additionally, Terminal-Bench wouldn’t be possible without its community. We’re so thankful to the over 1k members of our Discord who contributed and audited tasks, helped build and beta test Harbor, and made this such a fun project for everyone involved. https://lnkd.in/guZXbPVn https://lnkd.in/g7xqkjK8 https://lnkd.in/gETM-wkx

111

5 Comments