# A New Weighted Food CPI from Scanner Big Data in China

Zhenkun Zhoua,b , Zikun Songa,b and Tao Rena,b,* *Corresponding auther:
<EMAIL_ADDRESS>aSchool of Statistics, Capital University of Economics and
Business, Beijing, China
bChina Institute of Consumption Big Data, Capital University of Economics and
Business, Beijing, China

###### Abstract

Scanner big data has potential to construct Consumer Price Index (CPI). The
study introduces a new weighted price index called S-FCPIw, which is
constructed using scanner big data from retail sales in China. We address the
limitations of China’s CPI especially for its high cost and untimely release,
and demonstrate the reliability of S-FCPIw by comparing it with existing price
indices. S-FCPIw can not only reflect the changes of goods prices in higher
frequency and richer dimension, and the analysis results show that S-FCPIw has
a significant and strong relationship with CPI and Food CPI. The findings
suggest that scanner big data can supplement traditional CPI calculations in
China and provide new insights into macroeconomic trends and inflation
prediction. We have made S-FCPIw publicly
available111https://github.com/kayzhou/S-FCPIw and update it on a weekly basis
to facilitate further study in this field.

###### keywords:

CPI; Price index construction; Scanner big data; Retail sales

††articletype: ARTICLE

##  1 Introduction

Big data gives us an unprecedented opportunity to construct the real-time and
high-frequency economic indicators, which provides a novel perspective for the
economics research. The Consumer Price Index (CPI), being one of the key
economic indices, measures inflation by monitoring price changes in a basket
of goods and services commonly bought by households. It helps economists and
policymakers understand the market while prices are rising or falling over
time. However, China’s CPI is derived and computed through limited surveys,
and the figures are typically published on a monthly or quarterly basis.

Scanner data refers to structured information that includes product prices,
goods categories, sales volume and store location. This data is generated when
sellers scan the barcode of the goods and customers complete purchases at
retail stores. It not only provides new data for constructing price indices
but also enables indices at detailed aggregation levels [1]. The Netherlands
is pioneering the utilization of scanner data for CPI. Initially, researchers
constructed the price index just for coffee [2]. In Norway and New Zealand,
scanner data is officially used for CPI [3, 4]. Due to concerns regarding data
security and privacy, retail enterprises lack of motivation to provide scanner
data. Consequently, collecting and obtaining scanner data becomes very
challenging. Some researchers have turned to study available online price
data. Notably, there are two well-established research projects: the Billion
Price Project conducted by MIT [5], and the iCPI developed by researchers from
Tsinghua University for China [6].

To our best knowledge, there is currently no research that directly employs
scanner data to construct CPI or Food CPI in China. Consequently, we undertake
an analysis of scanner data from the China Ant Business Alliance, especially
for food prices. We establish an weighted index S-FCPIw and compare it with
existing price indices. This work paves a new way for real-time monitoring
change of price, reflecting macroeconomic trends and measuring inflation
inertia, and is a valuable supplement to the research of CPI.

##  2 Data

The data utilized to construct price index originates from the retail scanner
database provided by China Ant Alliance (CAA), which is a retail alliance
organization in China. The CAA has more than 100 member enterprises in 29
provinces in China, with a total annual turnover approaching 100 billion
Yuan222http://en.mayishanglian.com/. By utilizing the Alibaba Cloud platform,
the CAA has been collecting and processing the massive scanner data from the
member enterprises in real time.

Compared with survey, scanner data in our study offers three main advantages:
low cost, high frequency and rich dimensions. Here, we further elaborate on
the rich dimensions. The scanner database encompasses the sales data of 12
million products spanning 29 provinces in China. This allows us to construct
an index at city, province and nation level in terms of regional dimension.
Regarding the dimension of goods categories, it include 75 basic categories
and 14 sub-categories.

##  3 Method

We delineate the process of constructing S-FCPI and S-FCPIw below. Given that
both S-FCPI and S-FCPIw share the same calculations in the majority of steps,
we use the S-FCPI to denote both indices.

Classifying goods. The official criteria for goods classification, such as the
COICOP and the Statistical Report on Circulation and Consumer Prices in China,
are employed to reclassify the categories of goods in the scanner database
(Table 3).

Obtaining goods price. The selection of representative goods for each category
is facilitated by the local government based on sales volume and consumption
structure, thereby creating a relatively fixed basket of goods. In contrast,
S-FCPI incorporates almost all goods from the scanner data, utilizing the
product ID to correlate sales data across different periods. Consequently, we
create a variable basket of goods. Regarding the frequency of price
collection, the survey for CPI is gathered between one to five times per
month. However, each recorded purchase can be regarded as price collection of
the goods listed on the receipt. The availability of scanner data enables
real-time price collection, allowing the S-FCPI to be constructed not only
monthly, but also at a higher frequency.

Calculating weight. Since the categories of goods account for different
proportions, it is necessary to set corresponding weight for categories. The
weight in CPI is derived throughout the survey, and is slightly adjusted in
each year. In addition, CPI has no weight below the basic-category. The weight
in S-FCPI is calculated using real-time scanner data which enables timely
adjustment during the report period. Specifically, we select the corresponding
aggregate scanner data based on the time and region dimension, then calculate
the proportions of goods sales to determine the weights.

Calculating index. Step one, calculating the unit price. The scanner data
provides comprehensive information including both price and sales volume of
the stock keeping unit (SKU). By using the weighted average SKU’s price (unit
price) instead of the simple average used by CPI, we can accurately reflect
the differences between stores. We assign the share of sales volume of goods
as the weight and calculate the weighted average price across multiple stores.
$U_{k}^{t}$ represents the price of SKU $k$, $Sale_{k,e}^{t}$ represents total
sale volume at the store $e$, and $Q_{k,e}^{t}$ represents total sale amount
at the store $e$ and time slot $t$.

$U_{k}^{t}=\frac{\sum Sale_{k,e}^{t}}{\sum Q_{k,e}^{t}}.$ (1)

Step two, calculating price changes. The relative value $R_{k}^{t}$ of price
changes of each SKU in S-FCPI is calculated by comparing the unit prices of
the two slot $t$ and $t-1$.

$R_{k}^{t}=\frac{U_{k}^{t}}{U_{k}^{t-1}}.$ (2)

Step three, constructing the price index for the basic-category. In China, the
National Statistic Bureau adopts the fixed basket index theory [7] and uses
the Jevons price index, as China’s official CPI. It does not take into account
consumers’ preferences. Referring to CPI, we create the Scanner-based Food CPI
(S-FCPI). Regrading the basic-category $j$ and the sub-category $i$, S-FCPI is
calculated by

$\text{S-FCPI}_{j}^{t}=\prod_{k=1}^{N_{jk}^{t}}\left(R_{jk}^{t}\right)^{\frac{1}{N_{jk}^{t}}},$
(3)

where $N_{jk}^{t}$ is the number of goods in the category $j$. The above form
is often used when the weights are unavailable. The scanner data makes it
possible to construct the weighted price index for basic-category, considering
consumer preference. The Consumer Price Index Manual also points out that the
weighted index will be a preferable option for CPI [8]. Therefore, inspired by
the theory of aggregator function in Törnqvist index [9], we construct a new
weighted food index in China called S-FCPIw,

$\text{S-FCPIw}_{j}^{t}=\prod_{k=1}^{N_{jk}^{t}}\left(R_{jk}^{t}\right)^{\frac{S_{jk}^{t-1}+S_{jk}^{t}}{2}},$
(4)
$S_{jk}^{t}=\frac{{Sales}_{jk}^{t}}{\sum_{k=1}^{N_{jk}^{t}}{Sales}_{jk}^{t}},$
(5)

where $S_{jk}^{t}$ is the share of sales of $k$ in the basic-category $j$.
Step four, aggregate to the high-level index. China’s CPI uses the Laspeyres
price index for high-level aggregate, and its weights are based on the
household consumption from the low-frequency survey. Therefore, S-FCPI takes
the proportion of sales in the reporting period as the weight and employ the
Paasche price index to aggregate the index [10]. First, S-FCPI for the basic-
category at the city-level is aggregated at the sub-category city-level
S-FCPI, and then aggregated into the all-category city-level S-FCPI. Second,
we aggregate the index at basic-category province-level, sub-category
province-level and all-category province-level. Third, the S-FCPI is
aggregated at the nation-level for the basic-category, sub-category, and all-
category. Thus, S-FCPI and S-FCPIw are obtained across three region dimension
and three category dimension.

With the help of scanner big data, we can follow the method to obtain daily,
weekly and monthly S-FCPI and S-FCPIw. We further investigate the nation-level
all-category S-FCPI/S-FCPIw in the following.

##  4 Results

S-FCPI/S-FCPIw have significant advantages of low cost, high frequency and
rich dimension. Both indices can help to reflect macroeconomic trends, measure
inflation inertia and predict inflation. We examine the reliability of the new
indices by investigating their relationship with existing indices. The CPI,
FCPI (Food CPI) and FRPI (Food Retail Price Index) published by the National
Bureau of Statistics of China are selected to analyze and compare in our
study. We compare S-FCPI/S-FCPIw with the online food price index released by
the iCPI project.

###  4.1 Co-directional rate analysis

The index can be used to measure the change of price during the reporting
period compared to the previous period. To evaluate the ability of S-FCPIw to
reflect price fluctuations, we present the month-on-month indices in Fig. 1
from January 2021 to June 2023.

Figure 1: Month-on-month indices from 2021 to 2023.

To quantitatively measure the change direction of indices, we calculate the
co-directional rates and present the results in Table 1. The co-directional
rate is calculated by $=\frac{m}{M}\times 100\%$, where $m$ is the number of
months with the same change direction between indices, and $M$ is the total
number of months.

Table 1: Co-directional rate (%) between indices. | CPI | FCPI | FRPI | iCPI
---|---|---|---|---
S-FCPI | 68.97 | 86.21 | 69.57 | 72.22
S-FCPIw | 75.86 | 86.21 | 69.57 | 66.67
iCPI | 65.52 | 72.22 | 60.87 | -

The change direction between S-FCPI/S-FCPIw and FCPI are quietly close, with a
co-directional rate of 86.21%. They have the same change direction in 25
months and different directions only in 4 months. The co-directional rate of
iCPI and CPI is lower than that of S-FCPI/S-FCPIw. Besides, S-FCPIw and CPI
changed with the same direction in 22 months, accounting for 75.86%. The co-
directional rate between iCPI and CPI is the lowest (65.52%). The results
regarding FRPI are consistent with the above.

###  4.2 Correlation analysis

We further study the correlation between S-FCPIw and other indices. The index
growth rate is usually used to analyze the correlation between two chain
indices. Therefore, we compute the logarithmic growth rate of each indicator’s
chain index through ${x_{rate,t}}=\ln{x_{t}}-\ln{x_{t-1}}$ and then analyze
the Spearman correlation (Table 2).

Table 2: Spearman correlation between indices. Note: ∗∗∗, ∗∗ and ∗ are significant at the level of 1%, 5% and 10% respectively. The two rows for each pair of indices represent correlation coefficient and p-value. | CPI | FCPI | FRPI | iCPI
---|---|---|---|---
S-FCPI | 0.5009∗ | 0.5524∗ | 0.5804∗∗ | 0.5664∗
0.0972 | 0.0625 | 0.0479 | 0.0548
S-FCPIw | 0.6340∗∗ | 0.7063∗∗ | 0.7343∗∗∗ | 0.2867
0.0268 | 0.0102 | 0.0065 | 0.3663
iCPI | 0.2837 | 0.3776 | 0.3776 | -
0.3715 | 0.2262 | 0.2262 | -

S-FCPI demonstrates a strong positive correlation with CPI (coef=0.5009∗),
FCPI (0.5524∗), and FRPI (0.5804∗), indicating significant relationship
between S-FCPI and these indicators. Surprisingly, we find that the
correlation for S-FCPIw is more significant than that of S-FCPI. S-FCPIw
exhibits a strong positive correlation with CPI (0.6340∗∗), FCPI (0.7063∗∗)
and FRPI (0.7343∗∗). However, we do not find a significant relationship
between iCPI and other CPIs, including CPI, FCPI and FRPI. This implies that
iCPI could not reflect the change in CPI while considering the correlation.

##  5 Conclusion

In this study, we demonstrate the construction method of a new weighted price
index using scanner big data. Our work addresses the deficiency of China’s
CPI. Through co-directional rate and correlation analysis, we find that
S-FCPIw has a significant relationship with price indices in China. However,
S-FCPI has certain inherent limitations. Researchers can conduct further
research in these promising directions. Regarding the theory, we could explore
the issues of chain drift and transcendental index through scanner big data.
In terms of index construction, S-FCPI can expand categories of goods and
consider online price data. Based on the new index, researchers could further
study the measurement of inflation inertia, analyze the stickiness of goods
prices, and examine the heterogeneity of goods price fluctuating among
regions.

## Disclosure statement

The authors report there are no competing interests to declare.

## Funding

This work was supported by National Social Science Fund of China (Grant No.
19BGL076), National Natural Science Foundation of China (Grant No. 72002141),
Beijing Social Science Fund (Grant No. 20GLA074), R&D Program of Beijing
Municipal Education Commission (Grant No. KM202210038002), Special Fund for
Fundamental Scientific Research of the Beijing Colleges in CUEB (Grant No.
QNTD202109).

## References

  * Haan and Van [2011] J. D. Haan, H. A. Van, der Grient,  Eliminating chain drift in price indexes based on scanner data,  Journal of Econometrics 161 (2011) 36–46.
  * Haan and Opperdoes [1997] J. D. Haan, E. Opperdoes,  Estimation of the coffee price index using scanner data: Simulation of official practices,  Simulation of Official Practices”, Third Meeting of the International Working Group on Price Indices, Stats (1997) 16–18.
  * Nygaard [2010] R. Nygaard,  Chain drift in a monthly chained superlative price index,  article presented at the joint UNECE/ILOs workshop on scanner data, Geneva, Switzerland (2010).
  * Ivancic et al. [2011] L. Ivancic, W. E. Diewert, K. J. Fox,  Scanner data, time aggregation and the construction of price indexes,  Journal of Econometrics 161 (2011) 24–35.
  * Cavallo and Rigobon [2016] A. Cavallo, R. Rigobon,  The billion prices project: Using online prices for measurement and research,  Journal of Economic Perspectives 30 (2016) 151–78.
  * Taoxiong et al. [2019] L. Taoxiong, T. Ke, J. Tingfeng, Z. Li,  Design and application of novel cpi based on online big data,  Journal of Quantitative & Technological Economics 36 (2019) 21.
  * Qiang [2006] X. Qiang,  The theoretical framework of cpi: Fixed-fixed-basket index or cost-of-living index,  Research on Financial and Economic Issues (2006) 19–27.
  * Graf [2020] B. Graf,  Consumer price index manual, 2020: Concepts and methods,  in: Consumer Price Index Manual, 2020, International Monetary Fund, 2020.
  * Diewert [1976] E. Diewert,  Exact and superlative index numbers, journal of econometrics, vol. 4 (1976).
  * Diewert [1998] W. E. Diewert,  Index number issues in the consumer price index,  Journal of Economic Perspectives 12 (1998) 47–58.

##  Appendix A Supplementary material

Table 3: Food basic-category and sub-category for S-FCPI. sub-category | basic-category
---|---
Grain | Rice, noodles, other grains, and grain products
Tuber | Tuber, Tuber products
Bean | Dried beans, soy products
Edible oil |  | Rapeseed oil, soybean oil, peanut oil, sunflower oil,
---
camellia oil, blend oil, linseed oil, corn oil,
olive oil, butter, other edible oils
Vegetables and edible fungi |  | Onion, ginger, garlic and pepper, root vegetables,
---
mushroom vegetables, nightshade vegetables, leafy
vegetables, dried vegetables and dried bacteria
and products
Neat of animal |  | Pork, beef, mutton, other livestock meat and
---
by-products, livestock meat products
Meat of poultries | Chicken, duck, other poultry meat and products
Aquatic products |  | Fish, shrimp, crab, shellfish, algae, soft pod,
---
other aquatic products
Eggs |  | Eggs, duck eggs, goose eggs, quail eggs, pigeon eggs,
---
other egg products and products
Dairy |  | Pure milk, pure goat milk, yogurt, milk powder,
---
milk beverage, other milk products
Dried fresh melons and fruits |  | Fresh fruit, nuts, candied dried fruit,
---
other melon and fruit products
Candy and pastries |  | Sugar, sweets and chocolate, pastries,
---
other confectionery pastries
Condiments |  | Edible salt, soy sauce, vinegar, cooking wine,
---
chicken essence, monosodium glutamate, sesame oil,
seasoning sauce, chili oil, spices, other condiments
Other food categories |  | Convenience food, starch, and products,
---
puffed food, baby food