Application testing is a critically important step in software development. It is also, unfortunately, an increasingly difficult part of the development process to perform. The sheer number of different combinations of operating system and form factors means that anyone building applications for mobile devices literally needs to have a room full of different devices to test upon.
They could, of course, revert to a third-party Real User Monitoring (RUM) tool. Vendors like SOASTA and NewRelic offer developers the ability to assess how their application will perform on different devices without physically owning those actual devices. The rise of RUM services has, therefore, made developers' lives significantly easier.
And where a tool is being used by developers, Amazon Web Services (AWS) will soon follow. The biggest and most innovative public cloud vendor (as judged by analysts, customers, and, frankly, other vendors) is today announcing its so-called AWS Device Farm offering. AWS Device Farm is an automated testing service for mobile devices and covers Android and Amazon's own operating system, FireOS. All of this it does, as expected, in the cloud, and at a commoditized price. Of course, there is a huge flaw in the list of supported mobile OSes being supported, but more on that later.
I spent some time talking with Amit Patel, General Manager of AWS Device Farm, and Trent Peterson, Product Manager for AWS Device Farm, about the purpose of the product, what it's going to do, and where they see it going.
Patel started by giving a clear justification for AWS going down this path:
"Application quality is increasingly important. On the mobile side, the issues that customers face when running applications can lead them to seek alternatives. The permutations of different devices and operating system versions lead to increased cost and complexity in testing. All this means that manual testing of mobile applications is both expensive and time consuming."
Device Farm is AWS' solution to this. It provides a mobile testing service that is fast, flexible and, above all, cheap. Device Farm runs automated scripts across multiple devices at the same time. It then aggregates the reporting to give developers an overall view of their application performance. Developers upload their application and select their framework of choice (currently Device Farm supports Appium, Calabash, Instrumentation, uiautomator, and Espresso). Device Farm then runs the automated tests across multiple devices and customers, culminating in a single result.
Of course, the elephant in the room here is that most developers are building applications for not only Android and FireOS mobile devices, but also iOS and the web. The one big hole in Device Farm is that it currently does not support iOS or web application developers. I put this to Patel, who told me:
"We continuously evolve our services depending on customer feedback. The space is currently emerging, and lots of players are coming at the problem [of application testing] from different angles. AWS will have a fleet of devices and it will evolve over time. We decided to do Android first because the demand from our customers was so great to deal with a multitude of different devices, screen sizes, form factors. We plan to evolve the service and continue adding more and more devices and platforms over time."
Regarding the lack of iOS support, an AWS spokesperson had this to say:
"We don’t have details around timing for iOS support at this time. Customers have told us that their biggest challenge is testing their mobile apps across the large number of disparate Android devices and flavors of the OS. As with all our services, we will continue to deliver innovations to Amazon Device Farm, including support for other platforms, based on customer feedback and requests."
Hmmm…weird. Oh well, on to the economics. Device Farm is priced in AWS' usual per-minute method. In the case of Device Farm, AWS is pricing the product at 17 cents per device minute with the first 250 minutes of testing being free. Interestingly, and in what will be a boon to larger organizations, AWS is introducing unmetered testing at a cost of $250 per device per month.
Device Farm looks to be a very good initial foray into testing for Amazon. It doesn't go far enough, but given AWS' speed and appetite, I'm sure they have plenty of plans for it in the works.
This article is published as part of the IDG Contributor Network. Want to Join?