At Uber, we use robust data processing systems such as Apache Flink and Apache Spark to power the streaming applications that helps us calculate up-to-date pricing, enhance driver dispatching, and fight fraud on our platform. Such solutions can process data at a massive scale in real time with exactly-once semantics, and the emergence of these […]
Back when Jeff Bezos filled orders in his garage and drove packages to the post office himself, crunching the numbers on costs, tracking inventory, and forecasting future demand was relatively simple. Fast-forward 25 years, Amazon’s retail business has more than 175 fulfillment centers (FC) worldwide with over 250,000 full-time associates shipping millions of items per […]
Synced spotlights 10 artificial intelligence papers that garnered extraordinary attention and accolades in 2019. The volume of peer-reviewed AI research papers has grown by more than 300 percent over the past three decades (Stanford AI Index 2019), and the top AI conferences in 2019 saw a deluge of paper. CVPR submissions spiked to 5,165, a […]
What’s the best way to safely increase parallelism in a production Node service? That’s a question my team needed to answer a couple of months ago. We were running 4,000 Node containers (or ‘workers’) for our bank integration service. The service was originally designed such that each worker would process only a single request at […]
This is the story of how we built ctlstore, a distributed multi-tenant data store that features effectively infinite read scalability, serves queries in 100µs, and can withstand the failure of any component. Highly-reliable systems need highly-reliable data sources. Segment’s stream processing pipeline is no different. Pipeline components need not only the data that they process, […]
A new hire recently asked me over lunch, “What does DigitalOcean’s tech debt look like?” I could not help but smile when I heard the question. Software engineers asking about a company’s tech debt is the equivalent of asking about a credit score. It’s their way of As a cloud provider that manages our own […]
Boeing’s CST-100 Starliner launched into space for the first time today, but the spacecraft failed to make it into a stable orbit that would allow it to rendezvous with the International Space Station.What happened: An Atlas V rocket safely carried Starliner into space from Cape Canaveral Air Force Station on Friday, but the capsule had […]
At LinkedIn, we like to say that experimentation is in our blood because no production release at the company happens without experimentation; by “experimentation,” we typically mean “A/B testing.” The company relies on employees to make decisions by analyzing data. Experimentation is a data-driven foundation of the decision-making process, which helps with measuring the precise […]
Read the 2020 Cloud Report from Cockroach Labs, and learn which cloud platform performs best for transactional workloads across TPC-C, Network Throughput, CPU, and Storage benchmarks. If there’s one thing we’ve learned in our three years of benchmarking cloud providers on transactional workloads, it’s this: the results change often. Last year’s report showed AWS dramatically […]
Andrey Zolotov, Gideon Low present their journey of transition to distributed data processing using GemFire and the challenges faced along the way. Source: infoq