Tips on Loading and Real-Time Searching of Big Data Sets

At OpenLogic, we manage a lot of data because we track all the world’s open source software. With the scope of data we’re talking about, individual tables hold many terabytes of data, which you can’t fit into a relational database. We need real-time random access to a huge non-static set of data so we can scan an organization’s code to see if they’re using improperly licensed code. We also need to execute long-running, complex analysis jobs against the same database of code so we can analyze trends and spot relationships. The solution involves Hadoop, HBase, and Solr.