What is business intelligence?

In the course of developing a business, it’s quite likely that you’ll encounter certain decisions. It’s fundamental to ask yourself if these choices are influenced by personal experience, based on logic or is it based off a more well conceived plan? The role of business intelligence is to eliminate guesswork and complexity by harnessing data to make more informed business decisions. Data-driven business outcomes reduces postulation, bias and rather provides solutions based on solid theories, methodologies, processes, architectures, and technologies.

How can business intelligence be achieved?

Business intelligence within a business, specifically tech companies, typically includes the following components:

• Data Mining and Analytics: By adapting data mining tools and integrating business datasets to streamline the analysis process. The common form of presenting business intelligence is through data visualization for extracting insights and greater verification.
  

• Anomaly Detection: By using algorithmic machine-learning methods to identify aberrations or even minor blips in expected patterns in a virtually unlimited number of metrics in real time. For example, on a product level, it can identify the number of DAU, detect customer retention and financial risk control. On a business level, it can chart employee turnover and financial metrics.
  

• Data Services: Focusing on data output, self-service tools to perform queries for each department. Meta-data and rights management, forming data middle office, supporting the data needs of other products.

What are the daily tasks of data analysts/scientists?

To achieve “data-driven”, intelligent decision making. Effective data is the cornerstone of analytical mining.

Where does the data come from?

Incomplete statistics, a professional working in the data field spends 70% of his time on obtaining data. Depending on individual business needs and the type of product, we require all various kinds of datasets in order to obtain raw data.

• Data is usually generated internally within the business. For example, an e-commerce platform needs to tally the order, quantity and distribution of the commodity. Food delivery services need daily statistics on their orders, quantity, timings. Ride-hailing services generate data just off every single ride. Generally, all this data would be stored within the product universe. In order to optimize your analysis, you’d need to integrate your internal, real-time data with synchronization tools.

• With regards to user generated data, the behavioral trajectory of users in products often provides intuitive feedback on product decisions. Take for example a news platform, you can track the page views of popular news, or a heat map of where users click on specific webpage. Users inevitably leave a series of "footprints" in the process of use. Generally, you can obtain user behaviors through front-end management, back-end logging, etc. Hence, it is essential to accurately locate and detect user movements and be disciplined about recording it. Take payments for example, from the time an order is submitted to securing a successful payment, there are a number of points within that process where users can choose not to proceed. But this gives you a data point. Based on the behavior, you can draw user profile/persona.

• In terms of access to third party data, this can be obtained through data providers such as ad-click through rates etc. In addition, when data analysis has output capabilities, you can seize the opportunity to work with enterprises that lack independent analysis capabilities and complete analysis work for them.

How is the data analyzed?

After we have obtained the necessary data for analysis, completed data preprocessing, model construction, and warehousing into a unified data warehouse, how should a business tackle analyzing this data?

Analysis of baseline

Different departments within an enterprise usually have different indicators of their concerned about. The market may be concerned about channel conversion rate. Sales may be concerned about payment. Personnel may care more about success rate, turnover rate and so on. The first step for analysts is to provide key metrics that clearly reflect the current state of the company or the current state of the product and present it to decision makers in a timely manner.

Drill into Root Cause

Identifying the baseline of the indicators is still insufficient to deliver meaningful analysis or insights. We also need to know how the baseline was formed. Analysts need to review the entire process with decision makers, chart the full life-cycle of the data, and know why the baseline is the way it is.

For example, if the success rate of recruitment is relatively low, we should review the entire process and perform data funnel statistics on every stage, from resume screening, telephone interviews, face-to-face interviews and finally, entry. If the pass rate of resume screening is relatively high, but the success rate of telephone interviews is low, then it is very likely that the screening of resumes is too loose. This is insights that we can provide to HR.

For example, if we look at product user retention rate, if the 30-day retention rate is relatively low and the user loss is serious, then the user activity can be observed weekly, monthly for loss and retention rates. If the user's 7-day loss rate is relatively large, it is necessary to act fast at the corresponding time node to sustain active use of the product.

Forecast

In hindsight, we should also be forecasting ahead. Based on historical records, we can also predict possible trends so that decision makers can make sharper choices and from there, analysts can estimate the possible outcomes and prepare for it. This requires us to use statistical methods, machine learning and other complex algorithms to learn historical data and derive estimates.

For example, it was estimated that the annual volume of data collected from TMalls’ Single’s Day sales had the ability to predict customer fraud, and it predicted the expected loss of customers in advance.

Predicting the future helps decision makers understand risks ahead of time, seize opportunities, and get the best results.

How should one look at the data?

The deliverables for data analysis are typically statistical results that are aggregated in a data form. Through data visualization tools like MioTech, we can visually present data characteristics and patterns in multiple dimensions. In every report, it is essential that we present macro-statistics that have the ability to not only give an overview, but also drill down to a granular level within its dimension.

BAT (Baidu, Alibaba, Tencent) investment details from 2009 to 2019, source: MioTech AMI

Why is business intelligence and data analysis increasingly important in the financial sector?

The types of data services

In addition to empowering internal decision making, data analysis provides data support for the broader user community. Let’s use the tech industry as an example to discuss a few types of data analysis services.

• Analysis based on user data

Users create its own data history through in-app transactions, content creation and other browsing and clicking activities. Through multidimensional analysis and statistical analysis of historical data, companies can gain user characteristics and behavior insights. For example, Meituan or the Ele.me provide merchant services, WeChat for Work provides work summaries, Alipay provides annual statements, etc.

• Custom Analysis

When users are not satisfied with the fixed statistical analysis dimensions provided by the product, they can use the analysis service by providing a custom query. For example, Jira's JQL custom query allows high-level users to complete queries using the semantics of class SQL. Even with internal analysis resources, you can provide analysis services for specific customers, meet their additional needs and deliver them through periodic mail or reports.

• Data Analysis Service providers like SaaS

They do not generate data themselves, but can provide standard solutions for specific industries; like data access, data processing, and analysis of the entire data stream. For example, Google Analytics can provide general website traffic analysis.Growingio and Sensors Data,successful Chinese companies, focus on industry analysis scenarios such as Internet e-commerce. Users can choose to submit data to public network platforms or privatized services.

Data Analysts or Data Scientists? What’s the difference?

There may be a distinction between data analysts/data scientists when recruiting in larger companies, but in smaller companies the boundaries between the two are rather vague. There is a saying which I think is more accurate. These two job roles reflect the broader internal shifts towards specialization within traditional service enterprises as they enter into the big data era.

The skillsets you need to become a data scientist:

For traditional data analyst, we usually pay more attention to his industry knowledge, business knowledge and analytical skills. He needs to have a wealth of industry experience, can quickly locate the required indicators according to the needs of the department, skillfully using tools to produce the corresponding chart or insight.

In the era of big data, the magnitude of data has increased exponentially on the macro level, and the types of data have become complicated. This requires data workers to use big data tools to fully capture data from multiple data sources, consume massive amounts of data, and ensure stable output.

For each piece of data, we want to extract as much information as possible. Therefore, we can’t be satisfied with only counting the number of commodity reviews.We need to also conduct sentiment analysis, keyword extraction, and understand user feedback. Current statistics alone isn’t enough for us. We don't just need to know what users historically like the most, we also need to speculate what users might like in the future. It’s insufficient to soly analyze structured data, unstructured data like images, audio, video and geographic data are also essential objects of analysis.

Therefore, more skills are required of data workers today. But learning technical skills is a relatively easy process, especially in the era of big data. The emergence of various automated tools provides more convenience for data processing. While data sensitivity, industry understanding and product understanding takes more time to digest, they are still more valuable than skills.