This high-level business goal boils down to a number of technical requirements. It means we need to find ways of automatically evaluating results based on customer data and adaptations, and we need to do this in many different contexts.

One of our engineers, Gianmario Spacagana[1] took a fresh look at how to tune clustering algorithms. In this blog post, I will briefly introduce validation of clustering algorithms so that you can later more easily appreciate and understand Gianmario’s upcoming blog post.

The American writer Mark Russell once said “The scientific theory I like best is that the rings of Saturn are composed entirely of lost airline luggage.” I believe many developers and marketers feel just as lost when exposed to some supposedly great clustering. That is, it is very difficult to evaluate supposed optimal data clustering without understanding some basic concepts about how to evaluate and dissect clustering results. Lacking at least apprentice cluster validation  knowledge, confirming or disapproving an assertion of optimal data clustering is difficult and becomes more of a leap of faith than a reasoned judgment. I will attempt to give you enough information to ask the right questions.

Clustering involves grouping individuals in a population so that individuals in the same cluster share some properties of interest. Algorithmically, the goal of a clustering algorithm is to minimize or maximize some function related to the individuals in the same cluster. We call this the objective function. Frequently, we think of it as some type of distance between the individuals in a population. For example, let’s say we group customers based on their lifetime value. Once we have predicted the lifetime value for each customer we can use this to group customers based on these values and potentially other values such as gender, age etc. In addition to formulating an objective function, it is key for clustering that you pick the appropriate values related to the individuals to consider in your clustering. These values are often called the features.

There are many different algorithms for actually performing the clustering. Clustering algorithms are potentially very process intensive and the complexity grows quickly with the size of the data set. Each algorithm is attempting to deal with the complexity in a different way.  We will not delve into these methodological details, but if you are looking for a detailed technical survey of them, you can start with this paper by Jain et al.[2]

One common technique in clustering is to use some kind of heuristic that allows the algorithm to reduce the number of calculations. As an example, the K-MEANS algorithm guesses clusters as a starting point. It then uses this guess to reduce the computations, while hoping to converge on an adequate clustering by iterating and moving individuals between clusters as the structure evolves.

The key question I’d like to explore is how we can determine if a clustering is valid or not. The procedure of evaluating the result of a clustering algorithm is called cluster validity. There are three different perspectives on cluster validation: external, internal and relative. External involves comparing the result to some external pre-defined structure that reflects an intuition about what should characterize a cluster. It could be as simple as eying the cluster or it could be more sophisticated, using pre-labeled data.

Internal validation refers to some internal properties of the clustering. Compactness is one criterion, and it requires that clustered individuals be as close to each other as possible given an objective function. Separation is another criterion, and it requires that clustered individuals be as clearly separated from each other  given an objective function.

Gianmario employs a few different techniques in his internal evaluation, including AIC, David-Boudin and Silhouette. For external validation, he uses AdjustedRand, which is a statistical measure of cluster similarity. He uses that to compare clustering results with previously known clusters determined to be of a certain quality. This way, we achieved an external validation of clustering results.

So, when presented with a clustering, don’t worry too much about the algorithm. Instead ask:

1. What was the evaluation criteria used in validating the clusters. Were they internal or external?
2. What was the objective function of the algorithms?
3. What properties were considered for internal validation, and what were the results?
4. If external validation was done, what pre-defined structure or intuition was used?

If you are curious about the algorithm, ask about what assumptions or heuristics the algorithms uses. Each scalable algorithm makes some assumptions; the question is does it makes sense for your problem? We will expand on this in a future blog post.

During 2012, Gianmario Spacagana looked into auto-tuning of clustering. He titled his thesis “TunUp: A Distributed Cloud-based Genetic Evolutionary Tuning for Data Clustering.”  Gianmario was a student of Polytechnic of Turin and the Royal Institute of Technology on Stockholm. He will provide a more detailed summary of his work in an upcoming blog post. 

In the meantime, if you would like more information, you can read Gianmario’s thesis presentation, which he posted here: http://www.slideshare.net/gmspacagna. He and AgilOne also published the source code on GitHub if you are interested in looking into it in more detail: https://github.com/gm-spacagna/tunup.

Well, if you present a problem to Wharton’s Research Center for Customer Analytics, you won’t get an answer right away, but you’ll be guaranteed maximum brainpower and a decidedly objective view of the issue.

As pointed out by Stephanie Overby of DataInformed, Wharton’s Research Center for Customer Analytics offers a crowdsourcing-like approach to solving data-centric business problems. The “crowd,” though, is limited to academic experts in customer data analysis, so you will have a longer wait for an answer.

Launched by veteran marketing professors, Dr. Eric T. Bradlow and Dr. Peter S. Fader, the center has six dedicated staff members, a small team of research assistants and more than 1,100 scholars in its research network.

Research from Around the Globe

Here’s how it works: In exchange for providing access to a customer data set and making a financial contribution toward the center’s operating costs, Wharton’s Research Center develops a research project addressing a problem suggested by the sponsor and presents it to prospective researchers from academic institutions around the globe who then submit proposals. The sponsoring company selects six proposals and each research team devotes a year to a solution.

The Waiting is the Hardest Part

Overby quoted Tom Thomas, executive director of marketing intelligence for digital ad agency Organic, which leveraged its parent company Omnicom’s sponsorship with the center to help take the agency’s analytics to the next level, as saying: “Predicting, indexing, and more dynamic measurement of online consumer behavior requires advanced mathematical, econometric and statistical skills. It’s hard to find this combination of skills in other agencies or consultancies.”

For those marketers with time on their side, Wharton’s Research Center for Customer Analytics may be worth the wait. Visit its Lifelong Learning webpage for free video lectures by Wharton faculty members.

Of course, organizations looking for quicker solutions to their marketing analytics problems, can check out www.agilone.com and request a demo of our easy to use, ‘data scientist in the cloud’ solution.

Today that question couldn’t be more relevant when discussing the virtues of predictive analytics versus brand experience. An article on Content Equals Money points out the dichotomy between the two, and whether the collection of  “1’s and 0’s” should win out over observation of human interaction.

Predictive Models Result in Better Marketing Decisions

The article makes good cases for both sides. Predictive analytics allows companies to make better decisions on what marketing content to run, where to run it, and who to target it to.
Predictive analytics also helps suggest products customers might like, figures out what devices to optimize your campaign for, and devises the most effective calls to action for driving sales.

But Emotions Play a Huge Part in Marketing

Nonetheless, there’s a lot to be said for human interaction. Buying a product or spending money on a service is no longer a simple transaction based on a consumer seeing an ad, deciding to fork over some dollars, and getting a physical product in return.

Thanks in large part to social networks; the brand experience is a huge component of gaining loyal customers. Predictive analytics simply can’t account for a customer’s emotional response to a video that’s gone viral.

Predictive Analytics Drive Brand Experience

While the article concludes that predictive analytics and the understanding of the brand experience should both be employed for maximum effect, I believe that the answer is both-and, not either-or. Predictive analytics can help illuminate the brand experience.

Work we did at AgilOne for a market intelligence company bears this out. With a growing business, the company was looking for a better way to deepen engagement with its clients. Through predictive analytics, AgilOne developed a system that provided clients with timely and engaging content, customized specifically for them.

If that’s not enhancing the brand experience, I don’t know what is.

(Check out case studies on how AgilOne’s cloud-based predictive marketing has helped clients better understand their customers and lift the business bottom line.)

We are seeing massive shifts in consumer behavior. They are mobile, they are social, and they are impatient. We have to engage them on their terms, across channels. We have to leverage the data we have about them: not just big data, but what Forrester calls “right data.”

On top of this trend, we have the flood of information coming at us from sensors of all kinds. Some of these sensors customers are even willing to wear 24/7 to help them improve their health (Nike FuelBand, FitBit). But even setting aside these types of leading-edge consumers, we still need to find in all that data the right data that provides us with customer intelligence to drive engagement.

We have to overhaul how we think about recognition and identification of customers to tailor our messages. To quote Forrester, “Marketers need strategies and systems in place to follow their consumers as they flit from one channel to the next. CI professionals need to consider recognition – knowing who an individual actually is – together with broader customer context – what we can know about them, all in a privacy-compliant way.”

And we all know that there’s a sea of big data ready to help us, but how, in all of that, can we find the right data? Resources to deal with those exabytes of data are still beyond many of us. This is the elephant in the room when it comes to big data. But to be honest, this insight from the report did not discourage me at all. Although Forrester said, “the resources required to make those exabytes of data meaningful – data scientists, decisioning engines, and text and natural language processing tools – are still out of reach for many organizations,” here at AgilOne we have taken the power of machine learning and PhD analytics and packaged it for marketers just like you, and we stand ready to help you.

You don’t have to take it from me. You can download Forrester’s report and read it in its entirety.

Forrester’s scorecard helps you evaluate your email marketing program from a business process and a user experience perspective since both dimensions are key and offers recommendations for gauging your program’s performance.

Here are a few highlights. Shar suggests expanding your analytics criteria. Typical analytics criteria includes clicks, opens, and conversions. These are important, but Shar suggests supplementing them with more advanced analytics, creating a framework to determine the business value of email. The report states that “the best practice here is to measure email’s influence on a long-term customer loyalty and profitability.”

You’ll find high level information about topics like governance as well as key marketing topics like segmentation. There are also important down to earth tips such as making sure that the header invites opening. With the volume of email coming in, a tantalizing subject line is more important than ever.

You’ll also find gems like the following: “Successful email marketers combine preference-based segmentation with behavioral data; incorporate advanced email analytics with overall marketing analytics to optimize multichannel campaigns; and leverage long-term, profit-based metrics to benchmark their email programs.”

I found this report thought-provoking and helpful in my own work here at AgilOne and I hope you find it just as valuable.

In “The Importance of Social Activation and Going Beyond Last-Click Attribution,” Marcus Whitney, CEO of Moontoast, explains that these tactics leave a lot of meaning on the table. He instead recommends trying to understand the “social activation” of your customers.

Social activation is a much more thorough way to measure online revenue generation, because it takes a holistic view of social value across brand awareness, user engagement, and the use of multiple touch points. Whitney urges us to step away from “likes” and last-click attribution, and use something he calls the DITE Framework to drive engagement

• Discovery: Draw casual fans through building brand awareness
• Interaction: Create purchase intent with engaging posts
• Transaction: Drive purchases and email subscriptions by offering high customer value
• Endorsement: Facilitate advocacy with clear calls to action and sharing incentives

According to a Facebook / Datalogix study cited by Whitney, campaigns focused on maximizing reach through social activation saw a 70 percent higher ROI than those that only focused on clicks. That’s not to say I think that this is easy. Revenue attribution isn’t easy to track. This is why a two-pronged approach makes sense. Develop an engagement strategy such as the one Whitney outlines. To properly attribute revenue, however, you’ll need to consolidate all of the consumer’s behavior and better assess customer lifetime value. Technologies such as  machine learning can help you leverage that data across all channels and touchpoints and give you insight about where your revenue is really coming from.

Other key trends uncovered by Accenture’s survey included the rise of the Chief Data Officer. About 66% of the 600 organizations surveyed have appointed a CDO and of those who haven’t, 71% plan to in the near future. This certainly stacks up with what I have seen about the critical importance of data to companies large and small.

It’s not surprising, however, that companies are far from satisfied with their current use of analytics, with only 22% claiming that they are very satisfied. And all companies report an analytics skills gap: they need more analysts than they have. Accenture suggests building your analytics capability but notes that while data stewardship is hard to outsource—after all, who knows your data needs better than you do—getting help with predictive analytics and modeling is definitely an option. Here at AgilOne, we stand ready to help you expand your predictive analytics capabilities, offering you fresh insight with our business-friendly cloud-based solution.

And by the way, Accenture summarized some of its findings in a cool infographic, if you’re interested.

Davenport sees Analytics 1.0 stretching across a long time period –  about 1954 to 2009. It’s characterized largely by the use of internal data from traditional systems of record. Models, when they were created, took several months to generate. Analytics was a back office function and almost no one competed with analytics, a topic that Davenport wrote a groundbreaking book about in 2007. 

Around 2010, we enter Analytics 2.0, and we start hearing about big data. Companies start using Hadoop to process all that unstructured data and we see visual analytics begin to come to the fore, which he sees as a form of descriptive analytics. In this phase, descriptive analytics is still crowding out predictive and prescriptive analytics. Predictive analytics makes predictions about future events; prescriptive analytics uses business rules to suggest actions that we should take based on those predictions. Both of these types of analytics heavily leverage machine learning.  But I’m getting ahead of myself: that’s part of Analytics 3.0.

Analytics 2.0 also heralded a shift in the place of analytics in the organization. Analysts move closer to the business and competing with analytics becomes the norm. Data scientists are in high demand.

Lots of people would say we are still in Analytics 2.0, but Davenport suggests multiple reasons why he believes that we have entered Analytics 3.0. He gives several reasons:

• Companies are combining numerous internal and external data sources to drive predictive and prescriptive models
• Software that leverages agile analytical methods and machine learning is taking data from all of these sources and driving faster predictive and prescriptive analytics
• Analytics is playing a central role in driving strategy to the point that it’s entering the C-suite: take a look at how many organizations now have a Chief Analytics Officer

It’s an exciting and challenging time to be a marketer. We have so much data available; the potential is high for us to devise more effective marketing actions. But honing in on the most meaningful information to take just the right action is the key. Analytics 3.0 will take us there. And with all the attention coming from the C-suite, as Davenport says, we’ll actually compete with analytics.

Basketball great Michael Jordan said, “Talent wins games, but teamwork and intelligence wins championships”. I fundamentally believe that the way a company thinks about their way of working and the structure  of teams can either become a millstone around the neck or a competitive advantage. I received quite a few questions on how we develop software. So in this blog post I will talk about how we think about engineering teams at AgilOne for building marketing focused big data predictive applications.

As is illustrated in the “marchitecture” diagram below, the depth of AgilOne technology is extraordinary.  As a consequence there is tremendous variety and depth of skills that are required to deliver features in our applications. Each of our features require domain expertise in the specific predictive techniques we are developing. We have to make sure our calculated results are relevant and better than user’s intuition. The predictions need to be actionable and valid. Development also requires skills in data science and machine learning and an understanding of how to translate an idea into real algorithms. Implementing an algorithm requires very particular skills in understanding the limitations of data available as well as dependencies in data. In addition we need to address architectural objectives in making the solution scalable, robust, extensible, etc.  All features are exposed as functions in a highly graphical user interface, requiring skilled front-end designers and developers. In addition, we make features and predictions available through our APIs.

All these things taken together make it quite challenging to iteratively deliver enterprise-grade, easy to use, accurate, predictive features on a big data platform. But we have found a way.

How do we execute?

Like many other companies we use an agile methodology based on scrum . In fact we use scrum for all teams in our technology organization, including technical operations and IT. Each scrum team has the skills to deliver complete features end to end. As in many other engineering organizations this implies each team has an associated UI designer, front-end developers, and back end developers. For us it also means each team has knowledge in technologies such as Hadoop, hBase, R, Cascading, and OLAP. Last but not least our teams need to have the skill set of a data scientist. They need to understand the data, detect the patterns, and apply the right algorithms.

Within AgilOne we do have data scientists working closely with clients (who are all high-volume marketers with large data volumes) exploring what can be found out by analyzing their data. Their work is tremendously valuable for our customers and ultimately also for us. It allows us to learn what customers are interested in. We can generalize patterns and analysis across our client base. This results in great benefit for AgilOne as well as our customers.

In the engineering team we have taken the data scientist role one step further by defining a role we call Product Data Scientist. A product data scientist is a data scientist who is also part of an engineering scrum team. They perform generalized analysis that it can be applied across organizations in a multi-tenant cloud application. The product data scientist is a very rare individual. They must have a specific skillset, particular interests, and a unique personality. An AgilOne product data scientist is a world class mathematician with an analytical mind. But they are also a developer who can help turn an algorithm into a scalable working implementation. Some of our product scientists are mathematicians who have learned the art of software engineering while others are experienced software engineers what have had a lifelong interest in math and modeling. This role has enabled us to develop new analysis and models — and implement them within the same team — avoiding the “throw over the wall” problem.

To lead the delivery of these applications we have product managers that are extraordinary drivers. They are customer-oriented having worked directly with clients and have an analytical background and mindset.

I believe that our team set up has enabled us to deliver our predictive applications in a a more iterative fashion than a more traditional analyst v.s engineering structure. The reason is that teams have the analytical competency to understand where trade-offs can be made while keeping features useful and relevant. Since we removed the wall between data science and engineering we improve features and algorithms, and ship, very efficiently.

I have been amazed on how our team composition has multiplied the competency of each individual.  We have teams that can iteratively deliver advanced machine learning-based, scalable, multi-tenant predictive applications with great UI. The product data science role has worked out well. We have been experimenting with having product data scientists work alongside client data scientists. That cross pollination has been great as these individuals speak the same language which make sharing ideas and knowledge essentially effortless.

I believe we have turned the difficult problem of delivering functionality of extreme depth and breadth into a competitive advantage. 

Although often over quoted, the term Life Time Value (LTV) is one of the most underutilized but fundamental marketing concepts. The concept is based on the belief that the customer is the unit value for an enterprise and the sum total of customer value is equal to the enterprise value ( see Customer Equity).   How useful is this concept for those involved in the daily grind of running a marketing department?

We’ve been talking to several potential customers who have talked with marketing intelligence companies that provide on-demand LTV predictions that are updated nightly.  While most organizations are interested in understanding the future value of their customers, they don’t necessarily know how they’ll use LTV predictions because it can be interpreted in several different ways.

Value – can be revenue, margin, variable contribution margin and may include or exclude acquisition cost depending on the use case

LT – Lifetime :  can be historical, projected, upside etc.

Based on these definitions, LTV could be
1) LTV-Historical: Historical value of customer looking back since acquisition or last X months
2) LTV-Projected:  Projected value of a customer looking forward X years
3) LTV-Upside: Potential upside in customer value per year

These three different measurements/predictions can be utilized in different ways:

1) LTV-H:  This metric can be used to understand customer value trends, including whether a customer’s value is trending up or down, or if they are new/reactivated, or lapsed or inactive.  The change in LTV-H can be used to trigger lifecycle campaigns. 

2) LTV-P:  This metrics is best used for both acquisition source optimization and customer onboarding.  

• Acquisition Source Optimization: Based on short term LTV-P projections, a marketer can estimate which acquisition channels provide the best customers.  It is best to use short-term data for this calculation due to the changing nature of sources of acquisitions and offers.  If a marketer makes decisions based on projected LTV derived from old data, the calculations will not account for fast changing digital flows that happen in ad networks, keyword trends, and other modern acquisition channels.

• New Customer Onboarding: Another application of LTV-P is onboarding customers.  When a new customer is acquired, utilizing LTV-P and other observed and implicit properties of the customer (e.g., demographics, first order revenue, categories, email, web behavior, etc.), marketers should devise an onboarding process to ensure a successful second order from the customer.  Customers are usually very expensive to acquire, and more than 50% of new customers never come back to make a second purchase.  Successfully onboarding a customer could significantly increase LTV, impacting value in a compounding manner.  In other words, increasing new customer retention metrics will have a compounding downstream effect on the value of customers, and thus the value of the company.

3) LTV-U:  This is the most underutilized LTV metric.  Most organizations observe and predict value based on customer transactions.  However, very few companies calculate and utilize the upside potential of a customer.  Successfully calculating and operationalizing this metric will allow marketers to focus on the high potential customer within the context of acquisition, reactivation and growth objectives.
 
How to calculate LTV – components of the equation

LTV is defined as the net present value of profits from the customer.  Profit incorporates the following components that are the levers to increase the lifetime value:

• Number of Orders
• Products per order (product mix)
• Product Margin %
• Product Price
• Quantity per product
• Discount cost
• Promotion cost (cost to reach)
• Acquisition cost
• Retention rate
• Cost of capital

In various cases, acquisition cost can be included or excluded in the calculation depending on the use case.  In other words, if a customer is acquired, putting the customer acquisition cost in the equation will not be actionable, since this is already a sunk cost and we need to make sure for all these customers, we need to maximize the future profits.  Whereas if we’re trying to allocate resources for acquisition sources, we should include the LTV by source including the Acquisition cost, since we want to acquire customers who will deliver positive ROI within a certain period of time.  

It changes, keep up

The lifetime value of a customer, whether predicted or calculated, changes over time.  This change signals underlying trends, risks and opportunities.  Make sure you don’t have a static view of LTV calculations, but look at as snapshots in time and calculate transitions.  When a High LTV customer is trending to Low LTV over the past year, it signals a valuable at-risk customer.  You as a marketer need to focus on this.  Each customer lost, especially a high value customer is extremely expensive to replace.  

Conclusion
Focusing on LTV is very important, but using LTV to create actionable triggers is perhaps more important than knowing exactly what the LTV is.  Additionally, once triggers have been set, it’s the campaign effectiveness that allows a marketer to achieve customer objectives.  I’ll discuss campaign effectiveness in a separate post.

MOUNTAIN VIEW, Calif. – March 20, 2013 – AgilOne, the leader in cloud-based predictive marketing analytics, today announced that Saravanan Chettiar, formerly of Castlight Health and Salesforce.com, has joined the company as vice president of finance. Chettiar comes to AgilOne with more than 15 years of diverse finance and operations experience which will be invaluable in his new role leading the company’s finance, HR, and legal functions.

“AgilOne is at a significant growth stage coming out of our Series B round at the end of 2012. This is the year of growing sales and marketing and Saravanan’s finance leadership will prove invaluable in helping the company get to the next phase of success,” said Omer Artun, CEO of AgilOne. “His experience with some of the most successful SaaS companies in the world is an essential addition to our team. We are thrilled to welcome him to AgilOne.”

“AgilOne is dominating the market for predictive marketing analytics. Some of the biggest companies in the world are powering their marketing functions with AgilOne. I looked at many opportunities in the cloud computing space, and AgilOne truly stood out – both for the tremendous traction it has today and for the potential it has to extend its leadership in this space.  I am truly excited to join the team and to help lead the company’s financial and operational growth to the next stage,” said Chettiar.
Prior to AgilOne, Chettiar was vice president of finance at Castlight Health, a SaaS healthcare IT company with over $180M in capital raised.  Before that, he held leadership roles in the corporate planning and sales operations teams at Salesforce.com.  Chettiar has also held banking, public accounting, and internal audit roles at Credit Suisse, KPMG, and the Siemens Group. Chettiar has an MBA from the University of Michigan and a Bachelor’s in Accounting from Nanyang Technological University.
About AgilOne
AgilOne is a cloud-based predictive intelligence application for marketers. It is used by hundreds of brands including Shazam, Sports Authority, Moosejaw, ideeli, and shopPBS.org. AgilOne helps marketers connect with the individual in every customer. AgilOne prescribes exactly the right marketing offer, makes each relationship more profitable, and simplifies the science of marketing.  Based on a data-scientist approach, AgilOne makes big customer data clean and smart. Then, AgilOne recommends what immediate actions to take to increase revenue.  AgilOne is based in Mountain View, CA and is venture-funded by Sequoia Capital and Mayfield Fund. For more information, please visit www.agilone.com.

There is a lot of excitement about being able to introduce predictions and recommendations into applications. The goal is to improve the user experience, increase adoption and improve conversion rates. In some instances services want to analyze fairly structured data such as transactions and web-clicks. Other times the analysis concerns unstructured data such as text or short posts with their own “language” such as tweets. This has created an opportunity for cloud based Prediction API’s. In the last few years I have seen many different Prediction API’s appear, and some of them disappeared.

In this post I will argue that most prediction API are not as easy to make useful as may first appear. I am arguing that the “prediction” part is not the main challenge associated with the incorporation of prescriptive features into an application. I am not saying that you shouldn’t use these API’s but don’t expect the current breed of Prediction API’s to give you an out-of-the-box experience. Let me explain why.

Steps of a machine-learning process

When you compute predictions you normally use some sort of machine learning algorithm. These algorithms need good data to give good results. The GIGO idiom (garbage in, garbage out) is more applicable than ever. Supervised algorithms rely on good training data to create good models and provide adequate scoring. Unsupervised algorithms need features that are representative to work properly. The common theme is that garbage-data in results in garbage-predictions out.

The figure below illustrates the high level steps and algorithms that data typically must pass through to provide adequate results. [Admittedly somewhat of a simplification]

Data Quality is the step that involves taking raw data and cleaning so that we have a base data set that satisfies certain fundamental quality criteria.  If we are concerned with semi-structured data, quality may encompass things like making sure we have addresses that are valid and complete, names are sensible, numbers that are expected to be positive are positive, garbage rows are removed, etc. It can also include doing entity resolution and record linkage so that we get a more accurate picture of what is going on.

Data preparation is the next step in the process. And I consider this step somewhat model specific. By this I mean that what you do in this step depends largely on what type of model that you are trying to build. For example, in this step, we identify and remove outliers that will skew our model in addition we may normalize data. When records are missing data in specific fields we may try to approximate them. The goal is to have a representative data set that allows us to create as good a model as possible. Since models are also domain specific, I believe it Is safe to say that if you are building predictive models for a specific problem domain area you can do a better job preparing the data. This is simply due to the fact that you can make more accurate assumptions and hence prepare the data better.

Feature selection involves selecting the data from records that we use for the model building. The term feature refers to the attributes or fields of the data that are most relevant for the model we are building. A feature could be taken straight from the input data or be generated by a function applied to the feature. An example of a feature would be to decide that each noun in a text is a feature, a generated feature could be that we map words to their stem base. One goal is reduce dimensions to avoid redundant or interrelated features. We also do not want to include features that are not relevant to the model at hand. For structured data, the features may be specific attributes such as revenue, last purchase date, number of purchases etc. For unstructured data features are often words, stem base of words, n-grams etc.

Model Building, Model Selection and Scoring refer to the traditional steps of machine learning. When you look at software and literature related to machine learning, these are the steps they tend to focus on. Based on labeled data, features and possibly target variables (if supervised learning) we create a model that is later used for scoring.

I am sure you realize by now that if we get bad quality data in we will ultimately end up with models that will not perform well. Missing values or outliers can significantly impact models. Analysis of unstructured data will be impacted if we have a lot of bad data such as bad words, misspellings, and words that are not “real” words.

Predictive API’s

There is a long list of services that provide predictive API’s. Google Prediction API has been around for a while, and so has Saplo.com. Last year or so has brought some new ones such as BigML, Sociocast and PriorKnowledge[1]. I am sure there are quite a few more that I omitted [sorry]. AgilOne also provides a Prediction API, although with a different twist than many other vendors.

All vendors naturally provide various ways of uploading or referring to data.  Data can of course be in many different format  and at many different levels of quality as illustrated by the diagram below. For the purpose of this discussion I view it as a continuum from completely unstructured data such as text, to very structured data were we can expect a very well defined format and well defined manifest types. As far as I can see Prediction API’s tend to address one of two extremes. Either they process unstructured data such as texts of different sizes, providing tagging and various forms of text analysis. Or they focus on fairly structured data requiring that users have prepared and cleansed the data before uploading, or they are making predictions on very structured machine generated data such as web click data.

Big Data and machine learning are two hot areas that are driving loads of innovation. Some people think they need a lot of data to apply machine learning and others think they need machine learning to do anything with their Big Data.  But is this always true? You can learn a lot from your Big Data without complex machine learning algorithms; transformation of data and calculation of aggregates are two such examples. With that said, you can enable very powerful analysis by using machine learning techniques on large data sets. This post will help you better understand the relationship between Big Data and machine learning and when to use them together.

What do we mean by Big Data?

Big Data is a very active topic right now. We generate more data than ever before, and everyone agrees that we should use it to improve decisions, understand customers, detect system anomalies, etc. A study by the IBM Institute of Business Value [Analytics: the real-world use of big data] states that businesses associate Big Data with a greater scope of data collection and analysis. Companies also associate it with the techniques for handling massive amounts of data.

According to the IBM study, only 10% of companies say that Big Data is about large volumes of data. 18% of organizations consider Big Data to be related to an increasing scope of available information. 17% associate it with new kinds of analysis while 16% associate real-time analysis with Big Data. 

For the purpose of this post Big Data refers to a system’s ability to handle a dataset regardless of its size. These data sets can be huge: billions of purchase transactions or much smaller. With Big Data methods, we can process the data  regardless of size within a time that makes results practical and useful.

Learning is the key word in Machine Learning.

Machine learning refers to many different techniques, from the statistical, mathematical, and computer science communities. But at their core, these methods are all about creating data models that help us understand and predict the world. For example, we can create a language detection model using texts and their associated language such as English or Swedish. [training data]. Then, we can use this model to derive, with some acceptable accuracy, a specific text’s language. There are many Machine Learning algorithms. At the highest level we categorize them into supervised and unsupervised. A supervised model learns to predict a specific outcome using labeled data. For example, we build models that predict propensity to buy, using training data that labels people, which did and did not buy. Statistical regression is a typical example of a supervised learning algorithm. In contrast, an unsupervised algorithm finds patterns in your data, when that data does not have any labels. For example, we can build a customer segmentation model, when you do not know how many types of customers you have. But both supervised and unsupervised learning has two major stages: model building and scoring. Model building is when we create the model using training data, scoring occurs when we use the model to make a prediction about some new data.

What data is required to learn?

So what is the connection between machine learning and big data? Do you need big data to do machine learning? To cut to the chase: no. You do not always need big data. However, if you can apply machine learning to Big Data, then you can create some very powerful tools and insights you wouldn’t see any other way.

In many situations you can apply machine learning to a sample [subset] of the population that you care about. A sampled data set can be quite small and in many cases you can do the processing on a simple computer. This works when you are looking for a simple relationship such as normal distribution. For example, will one campaign return at least $10 more per customer, than another campaign? In this hypothetical situation, we only care about big effect sizes and so we should be able to reliably detect this effect in a small sample. If the effect size is small, we may not detect them, but it would be uninteresting anyway.

The key to good results: data, samples and effect size.

Problems occur when the distribution [in a statistical sense] we are trying to model is not a simple distribution. Another problem occurs if there is noise in the data. Noise makes it more difficult to detect patterns in the data and leads to the need for more data to create accurate models. For instance, noise could be that some  customers have missing or incorrect data.

An even bigger problem occurs if you create a sample with a bias, relative to the population. If the sample is biased, the model will be biased and of course lead to biased scoring. Bias is of course not a problem if we use the complete data set (you have all the data!). Lets assume we want to predict lifetime of people across the whole world. If we limited our training set to US population we would create a large, but biased training data set. If we used data from the whole world, we would more accurately be able to predict the lifetime of any newborn child anywhere in the world.

May we have some simple rules?

The table below provides a simplified synopsis of how Big Data and machine learning are related. In this context Big Data denotes the ability to apply machine learning to complete data sets.

If you are looking for a large effect size and for a simple relation, then using Big Data is an overkill. If the relations are complex or noisy you will benefit from using the complete data set [Big Data] but large samples may be sufficient. If you have biased samples you are in trouble, but Big Data can prevent this problem, while a large sample may give acceptable results in some situations.

One should note that even if model building can sometimes be done with sampling, in many situations scoring is still done on the complete data set. Scoring is normally less computationally intensive so the benefit of using subsets is diminished even if scoring on a subset would be make sense in a particular situation.

If you take away anything, take away this.

There is no direct dependency by machine learning on Big Data. Machine learning can be just as useful, even when you have a small sample of data as when you process a massive data set. In general, the ability to process Big Data will provide more accurate and powerful predictions and insights. Sometimes you will need Big Data to produce useful results due to issues with data, relations, effect sizes etc. but Big Data is not always required.

Correspondingly, think of Big Data completely independent from machine learning. For example, you might want to know how many of your customers have purchased items from at least three distinct product categories, no machine learning is involved in that calculation. Even for these types of data analytics, sampling can be useful. For example, let’s assume you want to know the average salary of individuals in a particular zip code. You can create a highly accurate estimate of this value, by calculating the average over a sample.

If you build your own data processing infrastructure you need to know how to deal with all this. When you can sample and when you really need Big Data. If you are one of our customers we are managing this for you. If  you are a Marketer you really should not need to know all this detail to become a data driven organization.

Thanks to Daragh Sibley for interesting feedback and discussions on the topic.

We’re so intimidated by unstructured data – fully 80 percent of all data – that we don’t spend enough time improving it. This prevents us from recognizing that our hands and our feet are attached to the same body. We don’t connect the well-organized transactional data we already have with the flood of new data. We’re so flummoxed by the big “Vs” – volume, variety, velocity – that only 0.5 percent of all data is being analyzed, according to IDC.

It’s time to shake off that mantle of fear. As Barrenecha says, “We have arrived at the point where we have practical solutions to address all aspects of enterprise information management.” Absolutely true. What we don’t always have is a CIO who takes a holistic view of setting big data strategy – a leader with vision is critical for leading us out of the valley of despair to implement not just centralized information management technology, but also policy that can prioritize data to be analyzed. That 0.5 percent analysis number sounds pathetic – and it is – but we certainly don’t need to analyze 100 percent of all the data we encounter – that’s not just noise; it’s a cacophony. We need to set priorities and collapse age-old obstacles between divisions. The technology for signal extraction is already here; we just need leaders who can reach across the aisles and get above the hype.

If we are going to take action on big data, we need to stop focusing on “data management” and start focusing on actionable events – Hollenbeck suggests all departments, not just marketers, funnel their data through an “Office of The Customer Experience.” These days every business unit from customer service to product management gathers information in its own way, and can’t see the big data picture. In each of these silos, the Voice of the Customer sings, but companies don’t hear it as harmony. Customers get frustrated because they want a company to listen to them across all of the channels they use. The best way to solve this problem is through a unified, company-wide effort to capture and analyze big data that really pays attention to the customer experience.

Don’t go. Universal plea from all marketers to our best customers. But wait – do we even know who they are? Do we have a great program to retain them?

Retention is a huge priority for us marketers. And a huge pain.

According to an RSR Research surveyfeatured in a MarketingCharts article, “Customer Retention and Loyalty Seen a Big Battle for Retailers,” we cited retention as our top marketing priority.

But the question remains, how exactly can marketers start to develop loyalty with our customers? According to the respondents (and at no surprise to the AgilOne team) predictive analytics is the most important weapon in retail retention. Marketers are starting to realize that customer demand is much more important than what we make or sell – 57% of respondents said customer purchase analytics are “very valuable.” But even more telling – and alarming – only 56% of the respondents said they knew who their best customers were, down from 73% in 2011.

In order to make leaps and bounds in customer retention – and start to really identify your biggest and most frequent spenders – marketers will need to push more heavily into predictive analytics. The RSR report goes on to say that 33% of respondents believed predictive analytics would be “very valuable,” while 26% had made a budget to build or acquire predictive analytics, and 23% were implementing or had already implemented a solution. Expect to see these priorities increase even more throughout 2013! So we can avoid, as Yaz says, the situation where “the doctor came too late”. 

 (image source)

How can that be? How can we engage effectively, without conducting analysis of every touch point, even the hard ones? Does this mean that marketers are de-prioritizing social media analytics because they have it all figured out? Or is it because the social media analytics they’ve attempted to date haven’t helped them optimize their engagement with customers and partners? Are the social media analytics these folks have investigated in a silo, or do they meld with other data sources? The Econsultancy “Quarterly Digital Intelligence Briefing” report says the consultants interviewed “more than 700 digital professionals” and have themselves concluded that “social media analytics as a focus is (worryingly) not on the radar as a priority for many companies.”

From this report, Olenski infers that although marketing optimization matters, few marketers know how they will go about it, which begs the question of whether they are finding marketing optimization tools to fill the gap between interest and action. Here at AgilOne, we have the answer!

Take a look at their conversation.You’ll see:  

• Omer’s inspiration for building AgilOne from the ground up
• How marketing is changing now that consumers are in the driver’s seat with purchases
• Omer’s background in machine learning and how it inspires our work in predictive analytics
• How AgilOne allows marketers to take meaningful action to generate revenue

Or, in Scoble’s own words: AgilOne’s CEO Omer Artun explains how it uses data management, marketing intelligence and predictive analysis to become a marketer’s solution and not just a tool. With its smart features, it provides marketers the ability to pinpoint their campaigns to the customers they really want.

What do you think of the interview? Let us know in the comments section! 
 

It’s even worse in the world of big data. Everyone is embracing it – but how fast can you get a meaningful answer?

First, a company needs to prioritize big data projects.

NewVantage Partners interviewed C-level executives at more than 50 enterprise-size companies and found that over 75%   were investing greater than $1 million per year on big data technology. Want to make your eyebrows raise even higher? 25% of large companies were investing more than $10 million! And according to a Computerworld/IDG News Service article on the study, “How to Use Big Data to Make Faster and Better Business Decisions,” their investment is paying off.  “They made it absolutely clear they could not do the programs they were doing without Big Data,” said Paul Barth, founder and managing partner of NewVantage Partners.So what exactly is the benefit behind embracing big data?

Second, how fast can you answer?

According to these surveyed companies, the real value comes from speeding up the time it takes to get to a business decision, or time-to-answer (TTA).

As Barth put it, “If your time-to-answer is 30 minutes in one case and 30 seconds in another, it really changes your business processes. It makes you much more effective as a business analyst.”  Of course, just having more data won’t speed up time-to-answer, and this has implications  across the organization. One flaw in thinking about big data is that it requires separate organizational responsibility and infrastructure. As if data is in a silo. Traditionally, organizations have separated analytics and business intelligence from data management. To improve TTA, the most cutting-edge companies are combining data management, analytics and business intelligence, and all data –  creating a big data “ecosystem”.

What is different about big data is that is can be explored as it arrives, without the need to clean, sort and write complex queries against it. Big data tells us what’s “new,” while enterprise data warehouses and traditional business intelligence tells us what’s “known.” We need both. Getting it to all work together will make us better marketers, and better businesses. And keep Whac-a-Mole in the arcade where it belongs!

Picking the right technology and approach to these problems is of strategic importance to us; hence we constantly look at how to evolve our infrastructure. It is nice for our customers, as they do not need to keep up with all the developments in this fluid space. Recently we looked at new ways of implementing our query services. We use it to allow customer to conducted domain oriented business intelligence queries at speed-of-though.

How you compare and contrast technologies depends on the problem at hand. In our case we looked at it from the perspective of speed-of-thought queries in SQL or a SQL “like” language, as well as the possibility of providing a pivoting UI on top of the query engine. We did not intend to satisfy other needs such as implementing machine learning or data cleansing with this solutions.

 The number of available technology offerings is growing day by day. Some solutions focus on providing very scalable analytics engines for very large datasets. Some focus on providing very fast query engines using columnar storage techniques. Other technologies try to provide both a NoSQL and RDBMS platform and solve the “thin pipe” problem. With the growing popularity of Hadoop, shuffling data between NoSQL stores (HDFS) and relational stores has become a bottleneck. The “thin pipe” refers to technologies like Scoop that provide a data transfer mechanism, but with relatively low transfer speeds. Other solutions provide massively scalable mechanisms for creating aggregated data allowing for quick queries, but requiring background aggregation processes. In addition traditional relational database vendors are combining columnar store or NOSQL implementations with traditional implementations of relations models so that enterprises can take advantage of both using one infrastructure component.

In our case we had additional consideration; being a cloud service we need to consider how well the solutions work for a multi-tenant solution storing lots of data. This is in fact a very significant consideration as it has both technical and economic implications.

 As a first step to help guide what to look for in these technologies, I propose you consider the following dimensions:

1)    How much data will you store? Is 100TB enough? Will you need several 100TB’s or do you need something that scales beyond that? For many enterprises 100TB is enough.

2)    If you need to scale beyond 100TB’s are you prepared to invest money and resources into developing sharding over multiple data stores? The alternative is to require that the solution automatically takes care of the scaling-out for you.

3)    What kind of query language do you need? Is it enough with a subset of SQL for simpler select queries or do you need the full SQL? Do you use tools that generate MDX queries?

4)    Do you want a commercial product or are you comfortable using an open source solution. If you are comfortable with an open source solution do you think you need commercial support?

5)    Is this strategic for you? and if so are you prepared to be an early adopter of a new technology that may give you an edge?

The diagram below informally illustrates some of the many aspects you may consider. Scale-up implies that the solutions architecture is not centered at a scale-out model. Super scale-out indicates the solutions main design center is around scaling to very very large data sets. The size of the circle indicates the price point, with more expensive offerings being larger circles than others.  The color indicates maturity: green indicating very mature, yellow early in life cycle, and red very early.

Things evolve quickly in the space of analytics databases right now. Solutions evolve and new solutions become available, and the strategies of emerging technology companies in this space are still evolving. You need to decide if this technology is strategic for your company and make your decision accordingly, and know what problem you are solving.