Nishant Gupta (hiring)

Two of AWS’ Sr. Principal Engineers, Nicholas Matsakis and Marc Bowes, take us inside the development of Aurora DSQL in a fascinating new blog. The post shows how the team scaled write operations without two-phase commit, overcame garbage collection hurdles, and ultimately embraced Rust for both data and control planes. What I find particularly interesting is how the team's journey with Rust evolved. They started cautiously, using it for a single component, and ended up rewriting almost everything in Rust. It's a reminder that sometimes the hard choices - like adopting a new language - can unlock tremendous value. The full post is well worth your time. It's not just about database internals, but also about team dynamics, a culture of learning (and questioning established approaches), and making difficult architectural decisions. Read it here: https://lnkd.in/eMhxBu9e Now, go build!

1,045

16 Comments

People often ask me about the differences in architecture between Amazon Dynamo (the 2007 SOSP paper), DynamoDB (the AWS serverless NoSQL database), and Aurora DSQL (the AWS serverless SQL databases). I captured the key points on durability, high availability, programming model, and trade-offs in my blog: https://lnkd.in/gkMG_tVD TL;DR: Despite the shared name, Dynamo and DynamoDB have little in common architecturally, and make very different trade-off decisions.

632

8 Comments

New year, and two new posts for my blog: In yesterday's post, I look the at the role of versioning in reducing coordination in distributed database systems: https://lnkd.in/gx9PQuxG. Reducing coordination helps latency, throughput, scalability, and availability, so this is a big win. In today's post, I look at a weird database isolation edge-case called Fekete's anomaly, and explore what it can tell us about isolation and coordination in database systems: https://lnkd.in/ge_XREph As I wrote these posts, I thought they would benefit from SQL syntax highlighting. I had no idea how to do it, and set myself a time budget of 5 minutes to figure out an OK solution. I asked Nova (via Amazon Bedrock), and got a simple recipe that works well in less than 60 seconds. Little things like this are the sweet spot of AI for me - I really enjoy being able to make things better with nearly zero effort.

241

5 Comments

In a new blog post, Marc Bowes looks at how updates work in Aurora DSQL, and what it means for scalable schema design. Read it here: https://lnkd.in/gmGEK8Dv A couple take-aways for application builders: - You mostly don't have to worry about hot read keys in DSQL, even in read-write transactions. DSQL can scale out per-key read throughput, and the design requires no co-ordination between transactions reading the same key. (Thank you, MVCC and Time Sync!) - Scalable applications do need to avoid high-contention write keys when doing updates. Writes need to be coordinated and ordered (to implement the I in ACID), which limits per-key throughput. - Most applications can avoid hot write keys with the right schema design. Most applications have already been written this way, because high-contention keys perform poorly on relational databases of all kinds.

239

11 Comments

Business processes are becoming complex enough that you can't run your company on just SAP, Oracle, and MuleSoft. Companies are investing heavily in AI workflows, which must integrate with legacy systems, existing microservices, and external API's. Open-source stacks are increasingly replacing the monolithic, proprietary stacks that were popular 5-10 years ago (especially before Kubernetes). These trends are only going to accelerate—and it means that you need to have a software engineering organization and architecture that is: - open-source - cloud-capable and cloud-agnostic - built for the demands of the modern enterprise. Colt McNealy and I have been working on such a platform, based on the open-source LittleHorse Kernel (and the surrounding Saddle ecosystem) which I am very excited about. You can check out the core Kernel on GitHub, and get in touch with Colt on Slack if you want to learn more! https://lnkd.in/gsEDVgDR

83

5 Comments

Since recently joining the Scale AI team as SVP of Engineering, I’ve been inspired by how deeply the team understands what it takes to operationalize AI at enterprise scale. Over the past several years, Scale has helped some of the world’s largest enterprises integrate AI across their most complex workflows. Today marks the next chapter in that journey. We’re excited to open-source Agentex, the agentic infrastructure layer in Scale GenAI Platform, built to enable enterprises to manage secure and reliable enterprise AI. We believe Agentex will become the standard layer for hosting and orchestrating AI agents by enabling developers to build freely while giving enterprises the control and reliability they need for mission-critical systems. Starting today, it’s open-sourced and available to everyone. Learn more about how Agentex powers enterprise AI workflows and try it out yourself: https://lnkd.in/gP7kQsMW

468

40 Comments

Today, Adobe announced the general availability of AI agents that will reshape how businesses build, deliver and optimize customer experiences and marketing campaigns. Powered by the Adobe Experience Platform (AEP) Agent Orchestrator, Adobe is also creating an AI platform for businesses to manage and customize agents from Adobe and across third-party ecosystems — ensuring agents can understand context, plan multi-step actions, refine responses and more. #AIAgents #MarketingInnovation https://lnkd.in/g8MQpwvw

284

2 Comments

🚀 A new kind of launch today: announcing https://cur.vantage.sh/ — a dictionary for AWS Billing codes that are present in Cost Explorer and the Cost and Usage Report! We profiled the number of distinct billing codes across our entire customer base and found that there are 60,000+ unique billing codes on AWS already and growing 🤯. We hear all the time that people are confused about the obscure billing codes present in AWS Cost Explorer or the AWS Cost and Usage Report. If you’re familiar, these are things like “Requests-Tier1” for S3 or “CW:GMWI-Metrics” for CloudWatch. There is usually really limited written resources for determining what these billing codes are even if you search around for them. As a result, we decided to build this new #FinOps microsite for looking up billing codes and attempting to explain in simplistic terms what these billing codes are. We have completed a number of services and the site is still in progress but we decided to put it live and start getting feedback. If you’re in the FinOps or Amazon Web Services (AWS) ecosystem, we'd love to hear your feedback.

155

17 Comments

LLMs are powerful, but they are stateless machines. They don’t carry memory or intent, they work with the context you provide at that moment. The same model can feel brilliant in one scenario and completely off in another. The difference is not the model, but the context and use case we design around it. The technology is already here. The real opportunity lies in how we shape it to solve real problems.

103

11 Comments

There's nothing more exciting than seeing world-class engineers innovate on behalf of customers. Here are three ways Snowflake just got smarter. ✨ Seamless Performance: Powered by Snowflake Optima Optima is our new intelligent optimization engine that automatically accelerates customer workloads. Automatic indexing and query optimizations happen behind the scenes with no knobs or extra effort. As a result, customers are seeing up to 15x faster queries on Gen-2 warehouses with zero additional compute or storage costs. Go here to learn more: https://lnkd.in/gxPBzhQZ 🧠 Empowering SQL Developers Details matter. We're making SQL more expressive, flexible and resilient. MERGE ALL BY NAME simplifies column mapping; updates and inserts happen automatically when names match. CONTINUE handlers let you handle exceptions without breaking flow. And Snowflake Scripting UDFs (now GA) bring full procedural logic directly into SQL. Go here to learn more: https://lnkd.in/gSvnnVew 🚀 Expanded Migration Support: Powered by SnowConvert AI Migrating from legacy systems just got easier. SnowConvert AI now supports more sources than ever including Teradata, Databricks, Oracle, SQL Server, Redshift, Azure Synapse, Google BigQuery, Greenplum, Postgres, Vertica, Hive, IBM DB2 and more. It automatically converts code, procedures and logic into Snowflake-native SQL, helping customers modernize faster and with confidence. Go here to learn more: https://lnkd.in/gX3TqdBT We're focused on making Snowflake faster, smarter and simpler every single day. No tuning. No hacks. Just better engineering under the hood, so builders can focus on building. #Snowflake #AIDataCloud #Optima #Performance #SQL #Migrations

278

4 Comments