How does a Prometheus Histogram work?

How does a Prometheus Histogram work? We looked previously at thecounter, gauge, and summary, how does the Prometheus histogram work? The histogram has several similarities to the summary. A histogram is a combination of various counters. Like summary metrics, histogram metrics are used to track the size of events, usually how long they take, via […]

Building a Service Mesh with Envoy

Service Mesh is the communication layer in a microservice setup. All requests, to and from each of the services go through the mesh. Also known as an infrastructure layer in a microservices setup, the service mesh makes communication between services reliable and secure.Each service has its own proxy service (sidecars) and all the proxy services […]

Monitoring blocked and passthrough external service traffic

What are BlackHole and Passthrough clusters? Understanding, controlling and securing your external service access is one of the key benefits that you get from a service mesh like Istio. From a security and operations point of view, it is critical to monitor what external service traffic is getting blocked as they might surface possible misconfigurations […]

Scaling a Mature Data Pipeline—Managing Overhead

Before delving into our specifics, I want to take a moment to discuss the technical stack backing our pipeline. Our platform uses a mixture of Spark and Hive jobs. Our core pipeline is primarily implemented in Scala. However, we leverage Spark SQL in certain contexts. We leverage YARN for job scheduling and resource management, and […]

Peloton – Uber’s Webscale Unified Scheduler on Mesos & Kubernetes

Mayank Bansal and Min Cai present Peloton, a Unified Resource Scheduler for collocating heterogeneous workloads in shared Mesos clusters. Its goal is to manage compute resources more efficiently while providing hierarchical max-min fairness guarantees for different teams. Peloton schedules large-scale batch jobs with millions of tasks and supports distributed TensorFlow jobs with thousands of GPUs. […]

Cloud-Powered, Next-Generation Banking

Traditional banks make extensive use of labor-intensive, human-centric control structures such as Production Support groups, Security Response teams, and Contingency Planning organizations. These control structures were deemed necessary in order to segment responsibilities and to maintain a security posture that is risk averse. Unfortunately, this traditional model tends to keep the subject matter experts in […]

The Future of League’s Engine

Hiya folks, Brian ‘Penrif’ Bossé, your local friendly Tech Lead of League here. I’m taking some time in between matches of TFT to wax philosophic about game engines and how we on League make decisions around what direction to take our custom game engine. Join me on a moderately long look at one dimension of […]

Operating Apache Kafka Clusters 24/7 Without A Global Ops Team

Earlier this year, the Streaming PubSub team at Lyft got multiple Apache Kafka clusters ready to take on load that required 24/7 support. The team’s operational burden for Kafka quickly started heading towards burn-out territory. On-call rotations started getting miserable because we’d get woken up at night due to failing hosts. Business requirements kept coming […]

Building a document understanding pipeline with Google Cloud

Document understanding is the practice of using AI and machine learning to extract data and insights from text and paper sources such as emails, PDFs, scanned documents, and more. In the past, capturing this unstructured or “dark data” has been an expensive, time-consuming, and error-prone process requiring manual data entry. Today, AI and machine learning […]

Introducing Maesh: A Service Mesh for Kubernetes

On September 4th, 2019, Containous, a cloud infrastructure software provider, released Maesh, an open-source service mesh written in Golang and built on top of the reverse proxy and load balancer Traefik. Maesh promises to provide a lightweight service mesh solution that is easy to get started with and to roll out across a microservice application. […]