サクサク読めて、アプリ限定の機能も多数!
トップへ戻る
世界禁煙デー
engineering.linkedin.com
Open sourcing Brooklin: Near real-time data streaming at scale Editor's note: This blog has been updated. Brooklin—a distributed service for streaming data in near real-time and at scale—has been running in production at LinkedIn since 2016, powering thousands of data streams and over 2 trillion messages per day. Today, we are pleased to announce the open-sourcing of Brooklin and that the source c
Editor’s note: The use of AI in LinkedIn products has been the subject of multiple press articles and research papers (some highlighted on this blog). With the release of a new LinkedIn Learning course about AI at LinkedIn, we asked our Head of AI, Deepak Agarwal, for a brief overview of what AI is and how it works, geared towards people who are interested in this growing field. In this post, we d
At LinkedIn, Kafka is the de-facto messaging platform that powers diverse sets of geographically-distributed applications at scale. Examples include our distributed NoSQL store (Espresso), stream processing framework (Samza), monitoring infrastructure (InGraphs), and derived data serving platform (Venice). Given these use cases, it’s not surprising that Kafka usage at LinkedIn has grown exponentia
CSS at Scale: LinkedIn’s New Open Source Projects Take on Stylesheet Performance Browsers use Cascading Style Sheets (CSS) to control the appearance of websites. From borders, fonts, and colors to layout, images, and animations, there are roughly 500 different style properties that can be declared with CSS. These properties are what make the visual diversity of the internet possible for your handh
Lighter than Lightweight: How We Built the Same App Twice with Preact and Glimmer.js The beauty of the web is that there is no “install” step. Someone, somewhere taps a link to your site, and moments later it appears instantly in front of them. At least, that’s the idea—but not all devices and networks are created equally. Sites that feel fast on a desktop computer with broadband can feel downrigh
Co-authors: Bruno Connelly and Bhaskaran Devaraj Being a Site Reliability Engineer (SRE) means having to talk about hard problems. Site outages, complex failure scenarios, and other technical emergencies are the things we have to be prepared to deal with every day. When we’re not dealing with problems, we’re discussing them. We regularly perform post-mortems and root cause analyses, and we general
LinkedIn recently released Project Voyager, our codename for the new version of our flagship application for Android, iOS, and mobile web. Voyager is the result of more than a year of product development work by over 250 engineers. We took the opportunity to rethink the LinkedIn experience from the ground up, not only from a product perspective, but on the engineering side as well. Before Voyager,
Application Pauses When Running JVM Inside Linux Control GroupsCauses and Solutions Linux cgroups-based solutions (e.g., Docker, CoreOS) are increasingly being used to host multiple applications on the same host. We have been using cgroups at LinkedIn to build our own containerization product called LPS (LinkedIn Platform as a Service) and to investigate the impact of resource-limiting policies on
Instant Messaging at LinkedIn: Scaling to Hundreds of Thousands of Persistent Connections on One Machine Coauthor: Cliff Snyder We recently introduced Instant Messaging on LinkedIn, complete with typing indicators and read receipts. To make this happen, we needed a way to push data from the server to mobile and web clients over persistent connections instead of the traditional request-response par
Don’t Let Linux Control Groups Run UncontrolledAddressing Memory-Related Performance Pitfalls of Cgroups Coauthors: Cuong Tran and Jerry Weng Summary The Linux kernel feature of cgroups (Control Groups) is being increasingly adopted for running applications in multi-tenanted environments. Many projects (e.g., Docker and CoreOS) rely on cgroups to limit resources such as CPU and memory. Ensuring th
Introducing {py}gradle, an open source Python plugin for Gradle Image Credit: Yiying Lu, Copyright: Gradle Inc. Gradle is a build automation tool that supports many programming languages. It coordinates the process of building software between multiple different code repositories and automates a number of important related tasks, like checking dependencies and warning programmers if something they
Open Sourcing Test ButlerReliable Android Testing, at Your Service Automated testing is a key component to LinkedIn’s 3x3 strategy for releasing mobile applications. As we developed the new LinkedIn Android app, we found that our tests had a major problem: our testing environment was unreliable, so our tests failed intermittently. We needed a solution that would let us rely on our tests to inform
Fast performance is a key feature of LinkedIn’s mobile applications. So when we first released the new LinkedIn iOS app, and we learned that our members were experiencing noticeable delays when viewing certain profiles, we immediately investigated the issue to find a solution. Performance profiling revealed that the main thread was spending a significant amount of time running Auto Layout. Auto La
We live in an age where we want to know relevant things happening around the world as soon as they happen; an age where digital content is updated instantly based on our likes and dislikes; an age where credit card fraud, security breaches, device malfunctions and site outages need to be detected and remedied as soon as they happen. It is an age where events are captured at scale and processed in
Open Sourcing Photon MLLinkedIn’s Scalable Machine Learning Library for Spark Machine learning is a key component of LinkedIn’s relevance-driven products. We use machine learning to train the ranking algorithms for our feed, advertising, recommender systems (such as People You May Know), email optimization, search engines, and more. For an in-depth example, check out these posts (part one and two)
Introducing and Open Sourcing AmbryLinkedIn’s New Distributed Object Store Media content has become ubiquitous around the web and almost all of Linkedin's new features interact with media in some form or the other. Profile photos, email attachments, logos, and influencer posts are a few examples of where photos, videos, PDFs, and other media types get uploaded and displayed to the end user. These
Designing SSD-Friendly ApplicationsFor Better Application Performance and Higher IO Efficiency These days, solid state drives (SSD) are being increasingly adopted to alleviate the I/O performance bottlenecks of applications. Numerous measurement results have showcased the performance improvement brought by SSD as compared with hard disk drives (HDD). However, in most deployment scenarios, a SSD is
Apache Kafka is a highly scalable messaging system that plays a critical role as LinkedIn’s central data pipeline. Kafka was developed at LinkedIn back in 2010, and it currently handles more than 1.4 trillion messages per day across over 1400 brokers. Kafka’s strong durability and low latency have enabled us to use Kafka to power a number of newer mission-critical use cases at LinkedIn. These incl
Open Sourcing Dr. ElephantSelf-Serve Performance Tuning for Hadoop and Spark We are proud to announce today that we are open sourcing Dr. Elephant, a powerful tool that helps users of Hadoop and Spark understand, analyze, and improve the performance of their flows. We first presented Dr. Elephant to the community last year during the eighth annual Hadoop Summit, a leading conference for the Apache
In modern data-driven businesses, the complexity that arises from fast-paced analytics, data mining and ETL processes makes metadata increasingly important. In this blog post, we share our own journey and a new open source effort that aims to boost productivity and data provenance. WhereHows, a project of the LinkedIn Data team, works by creating a central repository and portal for the processes,
Eliminating Large JVM GC Pauses Caused by Background IO Traffic Coauthor: Cuong Tran, Systems Architect In our production environments, we have repeatedly seen that applications running in JVM (Java Virtual Machine) occasionally experience large STW (Stop-The-World) application pauses due to JVM’s GC logging being blocked by background IO traffic (e.g., OS page cache writeback). During such STW
Open-sourcing PalDB, a lightweight companion for storing side data LinkedIn’s data products give our members recommendations, analytics and insights. As we continue to invent more products that leverage data for our members, we need to push the envelope in our data processing capabilities and do more with less. One issue that often comes up is what to do to improve the usability and memory efficie
Background LinkedIn’s FeatureFu project is a new open source toolkit designed to enable creative and agile feature engineering for most machine learning tasks such as statistical modeling (classification, clustering, and regression) and rule-based decision engines. In this blog post, we will detail the design and implementation of Expr in FeatureFu, provide examples of how feature engineering is b
LinkedIn started in 2003 with the goal of connecting to your network for better job opportunities. It had only 2,700 members the first week. Fast forward many years, and LinkedIn’s product portfolio, member base, and server load has grown tremendously. Today, LinkedIn operates globally with more than 350 million members. We serve tens of thousands of web pages every second of every day. We've hit
At LinkedIn, we are always looking for the best software development frameworks and tools to build great products. During our 11-year history we have adopted many web frameworks for UI development - among them Grails, Frontier (Linkedin's internal web framework), and most recently, Play. We love Play and are excited to increase our adoption of it across the company. As we evolve and scale our inte
Last fall we introduced Pinot, LinkedIn’s real-time analytics infrastructure, that we built to allow us to slice and dice across billions of rows in real-time across a wide variety of products. Today we are happy to announce that we have open sourced Pinot. We’ve had a lot of interest in Pinot and are excited to see how it is adopted by the open source community. We’ve been using it at LinkedIn fo
One of the responsibilities of the Data Infrastructure SRE team is to monitor the Apache Kafka infrastructure, the core pipeline for much of LinkedIn's data, in the most effective way to ensure 100% availability. We have recently developed a new method for monitoring Kafka consumers that we are pleased to release as an open source project - Burrow. Named after Franz Kafka's unfinished short story,
Open sourced software is no longer restricted to addressing small, low-level problems and for companies that can’t afford to built it themselves. Today it is backed by a mature community that creates and adopts it on behalf of some of the world’s biggest technology companies. Why the shift? Why should companies focus some of their efforts on open source? Why open source? It all starts by making yo
Optimizing Java CMS garbage collections, its difficulties, and using JTune as a solution Introduction Let's say you are running Java in a production environment, and one of the instances that plays a critical role fell out of SLA. What do you do? Java tuning has a history of shotgun-approach techniques to recover the service, but what if you had a tool that could: Tell you what the JVM is doing in
次のページ
このページを最初にブックマークしてみませんか?
『LinkedIn Engineering』の新着エントリーを見る
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く