Processing large data payloads used to require significant hardware investment. The Cloud opens new opportunities to perform such tasks with inherently different cost and scale characteristics. 47Lining has helped customers in energy, media & entertainment and life sciences process large data payloads at scale.
CureMetrix analyzes petabytes of radiology imagery to detect cancer and notify doctors that additional analysis is required. Their process relies on the ability to apply proprietary machine learning algorithms to extremely large scale data sets. They were attracted to the price / performance and horizontal scaling AWS could offer, but were not sure how to get started with a small team. 47Lining partnered with CureMetrix to securely ingest large data sets of imagery, apply CureMetrix algorithms to process them at elastic scale using Elastic Beanstalk and securely store the results in S3. The solution included:
- Launch of a secure, scalable, redundant environment using our open source DevOps tool called Nucleator. After minimal configuration, Nucleator launched VPCs with public and private subnets, Bastions, Nats, as well as an Elastic Beanstalk.
- Development of a controller application that can take medical imagery from a securely encrypted S3 Bucket and send a request for processing to an SQS queue.
- Creation of an image processing app, which runs on a worker tier in Elastic Beanstalk. The workers perform image processing algorithms on requested images and place the results in another S3 bucket. Spot instances can be leveraged to achieve attactive cost savings.