This event has ended. Visit the official site or create your own event on Sched.
Get your ticket and additional info and support at jcon.one
Back To Schedule
Wednesday, October 6 • 18:00 - 19:00
Object Compaction in Cloud for High Yield

Log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.
In file systems, large sequential writes are more beneficial than small random writes, and hence many storage systems implement a log structured file system. In the same way, the cloud favors large objects more than small objects. Cloud providers place throttling limits on PUTs and GETs, and so it takes significantly longer time to upload a bunch of small objects than a large object of the aggregate size. Moreover, there are per-PUT calls associated with uploading smaller objects. In Netflix, a lot of media assets and their relevant metadata is generated and pushed to cloud. Most of these files are between 10s of bytes to 10s of kilobytes and are saved as small objects on Cloud. In this talk, we would like to propose a strategy to compact these small objects into larger blobs before uploading them to Cloud. We will discuss the policies to select relevant smaller objects, and how to manage the indexing of these objects within the blob. We will also discuss how different cloud storage operations such as reads and deletes would be implemented for such objects. This includes recycling blobs that have dead small objects - due to overwrites, etc. Finally, we would showcase the potential impact of such a strategy on Netflix assets in terms of cost and performance.

avatar for Tejas Chopra

Tejas Chopra

Technical Leader, Netflix, Inc.
Tejas Chopra is a Senior Software Engineer, working in the Data Storage Platform team at Netflix, where he is responsible for architecting storage solutions to support Netflix Studios and Netflix Streaming Platform. Prior to Netflix, Tejas was working on designing and implementing... Read More →

Wednesday October 6, 2021 18:00 - 19:00 CEST
#4 Cloud Platforms