“Amazon Kinesis Data Streams processes multiple terabytes of log data each day, yet events show up in our analytics in seconds,” says Bennett. Typically, about 1,000 Amazon Kinesis shards work in parallel to process the data stream. Netflix’s Amazon Kinesis Data Streams-based solution has proven to be highly scalable, each day processing billions of traffic flows. “When it comes to our networking data, it’s more cost efficient to be able to scale up and down, which is not as easy to do with alternatives to Amazon Kinesis Data Streams,” says Bennett. The scalability of Amazon Kinesis Data Streams was a good fit for the Dredge application because of the cyclical and elastic nature of network usage at Netflix. Now we can focus on getting insights from the data rather than simply getting access to it.” “It took just a few API calls to centralize multiple terabytes of flow logs into Amazon Kinesis Data Streams. “It would have been daunting to publish, stream, and consume that much information from an external system such as Kafka,” says Bennett.
NETFITS UTILITY WINDOWS
Netflix uses the OLAP querying functionality of Druid to quickly slice data into regions, availability zones, and time windows to visualize it and gain insight into how the network is behaving and performing.ĪWS was the logical choice for Dredge in part because the data was already resident in the AWS Cloud. The enriched data lands in an open-source analytics application called Druid. “Dredge joins the flow logs with application metadata as it streams and indexes it without using a database, which eliminates a lot of the complexity.” “Usually, we would put the data into a database, which would build an index to enable faster querying,” says Bennett.
NETFITS UTILITY FULL
The application reads the data from Amazon Kinesis Data Streams in real time and enriches IP addresses with application metadata to provide a full picture of the networking environment. The solution Netflix ultimately deployed-known internally as Dredge-centralizes flow logs using Amazon Kinesis Data Streams. “We experimented with multiple designs and used many AWS products to get here.”
“Early in the design process, the flexibility to try different ways of processing the data was important,” says Bennett. Netflix set out to establish a new data source that could give it more insight into communication among applications and regions by combining VPC flow logs with application metadata.Ĭentralizing Flow Logs Using Amazon Kinesis Data Streamsįrom the outset, AWS enabled Netflix to experiment with different approaches to analyzing its network data.
NETFITS UTILITY HOW TO
“We’d know an application was connecting to others, but we didn’t know both sides of the conversation and how to optimize those communications or the placement of the applications on the network.”
NETFITS UTILITY SOFTWARE
“The data sources we had before we took on this initiative were one sided,” says John Bennett, senior software engineer at Netflix. However, in a dynamic environment like the one at Netflix, where an IP address can float between applications from day to day or even minute to minute, IP addresses alone don’t have much meaning. The company would also be able to increase uptime by quickly detecting and mitigating application downtime.Įach log record carries information about the communications between two IP addresses. This would enable Netflix to identify performance-improvement opportunities, such as identifying apps that are communicating across regions and collocating them. In particular, Netflix needed a solution for ingesting, augmenting, and analyzing the multiple terabytes of data its network generates daily in the form of virtual private cloud (VPC) flow logs. Monitoring and optimizing its network is critical for Netflix to continue improving customer experience, increasing efficiency, and reducing costs. This results in an extremely complex and dynamic networking environment where applications are constantly communicating inside AWS and across the Internet.
Netflix uses Amazon Web Services (AWS) for nearly all its computing and storage needs, including databases, analytics, recommendation engines, video transcoding, and more-hundreds of functions that in total use more than 100,000 server instances on AWS.
0 kommentar(er)
