Although some organizations have balked at using agile in a big data environment, our experience suggests that it is not only workable, it is critical. Resistance to agile reflects problems that some organizations have had when attempting to use agile in traditional enterprise data warehouse projects; i.e., projects not involving big data technologies. Three major problems make agile largely unworkable in that environment: (1) data modeling isn't easy to do iteratively; (2) excellent metadata is required up-front; and (3) accommodating requirements changes is difficult when these changes affect the data model.

My recent white paper, "Agile and Big Data: Sprinting to Business Insight," notes that many organizations have moved beyond traditional EDWs and embraced big data technology and modern data architectures, making it more feasible to adopt agile methodologies such as scrum and kanban. Our conclusion: If agile isn't the standard in your big data ecosystem, you're probably heading for serious trouble.

Click here to download "Agile and Big Data: Sprinting to Business Insight".