Besides being treated to research about why companies get different returns from similar IT investments, IT executives who attended the recent 2004 MIT Sloan CIO Symposium were treated to panel discussions on everything from enabling the agile enterprise to evaluating emerging technologies (see column).

Regarding the latter, several executives shared their thoughts on how to get this right. Steve Flammini, CTO at Partners Healthcare in Boston, said, "We define success before we pilot the technology, then test afterward." That way you have a ruler to gauge how far you have come.

Jim Medeiros, vice president of I.S. Shared Services at UPS, said his company has a formal evaluation process, "but there is always the danger that users get ahead of evaluators. Sometimes the evaluation isn't even done, but business units are already using the technology. If you don't follow the rules, it's anarchy. But if you follow them too closely you can get out of step with business needs."

Flammini echoed that. "If the process is too rigid, you stifle innovation. We do a lot of pilots. That gives us a handle on things like scalability and deployability. And it also gives us a sense if the ROI is there."

Panel member Robert Walmsley, president and CEO of Cape Clear Software, offered the outsider's view. As a small firm looking for the attention of big companies, he said "too many processes are set up to avoid risk, rather than identify innovation. Do you want people in the advanced technology group with a retail banking background or a venture capital background?"

GM's Chief Architect Richard Taggart was at the event to participate on another panel about emerging technologies, but spoke privately with Network World about agility. "Standardization and good architecture are the keys," Taggart said.

Regarding the former, GM has reduced the number of applications it uses from 7,000 three years ago to 3,000 today. (The company has also reduced its IT budget from about $4 billion per year to $3 billion.)

And GM is working with the Enterprise Architecture Interest Group to develop architecture standards that should make it more fleet of foot (see story). "Think of them as a set of reusable building blocks that can be tied together in different ways to address common business objectives like reducing cycle times or speeding decision-making," Taggart said. The first deliverables are due in June.