What Problem does Cosync Storage solve?

MongoDB Realm is perhaps the leading offline-first real-time data base for collaborative programming in the world today. It is the only truly scalable solution for client side data synchronization across a wide variety of platforms including iOS, OS/X, Android, Windows, and Web that runs on top of a unified server-less backend.

For collaborative programming however, data synchronization is usually only half of the challenge. Typically, an application includes both data and assets that need to be shared between users. Assets are defined as large immutable files such as images, music, videos, and/or documents. Currently, MongoDB Realm has a 4MB limit on the maximum object size supported — meaning that it really does not provide a solution for asset management (even if they are base 64 encoded). This is not to say that assets could not be broken up into multiple 4MB objects, but this would require additional management code within the client application to achieve. Lastly, because MongoDB Realm provides no video or audio streaming capabilities - an asset chunking strategy would probably not work.

Because of these constraints, most application developers resort to using an Asset Storage service like Amazon AWS S3 in addition to MongoDB Realm to solve the storage problem. Again, this requires additional client-side code to adequately implement a cross platform solution. More importantly, it often requires storing sensitive Storage Service credentials within the client code - which is an additional security risk.

Presently, the MongoDB Realm engineering team has tried to address this issue through a Third-Party service called AWS S3 Service Snippets. This solution however requires the application to upload the asset data to the MongoDB Application server, which in turns marshals it on to the Amazon S3 service. But again, there is a 4MB limit on the asset storage size, because of the base 64 encoding! Also, the very concept of having to upload an asset first to a MongoDB Realm server before it gets uploaded to a Storage Service is somewhat inefficient, as there is no inherent reason not to upload it directly from the client device to S3.

From an application developer’s point of view, an asset needs an HTTPS URL (universal resource locator) to be properly handled within a client program. This is because image caching services (like Kingfisher) usually require URLs to operate on. In addition, for image data, it is often necessary to handle different size cuts of the image (small, medium, large, and original) for different performance scenarios. Support for image cuts of varying sizes should be included in any asset management strategy. Lastly, video and audio assets require that the Storage Service be able to stream the data within an application. This can only be done through URLs to a Storage Service that supports CDN type functionality - like Amazon S3.

As far as MongoDB Realm is concerned, asset handling requires data structures to keep track of the asset within the Storage Service, and include functionality to compute write URLs to upload the asset data. It should also provide support for image asset cuts and video previews. The Storage service provides all of this functionality to a developer in one easy package that seamlessly integrates with any MongoDB Realm application.