#  PAE: LLM-based Product Attribute Extraction for E-Commerce Fashion Trends

Apurva Sinha  Walmart Global Technology
<EMAIL_ADDRESS>Ekta Gujral  Walmart Global Technology
<EMAIL_ADDRESS>
###### Abstract

Product attribute extraction is an growing field in e-commerce business, with
several applications including product ranking, product recommendation, future
assortment planning and improving online shopping customer experiences.
Understanding the customer needs is critical part of online business,
specifically fashion products. Retailers uses assortment planning to determine
the mix of products to offer in each store and channel, stay responsive to
market dynamics and to manage inventory and catalogs. The goal is to offer the
right styles, in the right sizes and colors, through the right channels. When
shoppers find products that meet their needs and desires, they are more likely
to return for future purchases, fostering customer loyalty. Product attributes
are a key factor in assortment planning. In this paper we present PAE, a
product attribute extraction algorithm for future trend reports consisting
text and images in PDF format. Most existing methods focus on attribute
extraction from titles or product descriptions or utilize visual information
from existing product images. Compared to the prior works, our work focuses on
attribute extraction from PDF files where upcoming fashion trends are
explained. This work proposes a more comprehensive framework that fully
utilizes the different modalities for attribute extraction and help retailers
to plan the assortment in advance. Our contributions are three-fold: (a) We
develop PAE, an efficient framework to extract attributes from unstructured
data (text and images); (b) We provide catalog matching methodology based on
BERT representations to discover the existing attributes using upcoming
attribute values; (c) We conduct extensive experiments with several baselines
and show that PAE is an effective, flexible and on par or superior (avg
$92.5\%$ F1-Score) framework to existing state-of-the-art for attribute value
extraction task.

###### Index Terms:

Attribute Extraction, PDF files, Bert Embedding, Hashtag, Large Language Model
(LLM), Text and Images

##  I Introduction

Assortment planning for future products plays a crucial role in the success of
e-Commerce as a platform. It involves strategically selecting and organizing a
range of products to meet customer demands and maximize sales. This process
involves analyzing market trends, customer preferences, and competitor
strategies to identify potential gaps and opportunities. By carefully planning
the assortment, retailers can ensure they offer a diverse and relevant range
of products that cater to different customer segments. This helps in driving
customer satisfaction, increasing sales, and staying ahead in the competitive
market. Walmart collaborate with trend forecasting company that provides
insights and analytics for the fashion and creative industries. They do not
release public reports, as their insights are provided through a paid
subscription service. However, they often share snippets of their forecasts
via blog posts or on social media. For example, they might report on upcoming
color trends for a particular season, predict consumer behaviors, or identify
emerging fashion trends in different regions. The trend forecasting company
also provides reports on retail and marketing strategies, textiles and
materials innovations, product development and lifestyle and interiors trends.
Their reports are typically used by retailers and marketers to plan and
develop their products and strategies.

Informal Problem 1. Given a set of target attributes (e.g., color, age group,
material), and unstructured information in the form of text and images: how
can we extract values for the attributes? What if some of these attributes
have multiple values, like colors or age group?

Correct predicted attributes helps in improved catalog mapping, which helps in
generating search tags on better content quality of products. Customers can
filter for products based on their exact needs and compare product variants
promptly. Resulting in a seamless shopping experience while searching or
browsing a product on an E-commerce platform. The Product attribute Extraction
(PAE) engine can help the retail industry to onboard new items or extract
attributes from existing catalog.

Figure 1: Example of Text and Images for Attribute Extraction Figure 2: End to
End PAE mapping to product catalog

Motivating Example Retailers can use upcoming market trends to decide on
product catalog assortment planning based on upcoming seasons like spring,
fall and summer. For a concrete example, refer to Figure 1, a classic shirt
(with unstructured text and image data), it talks about peplum or ruffle hem
being used on Global Organic Textile Standard (GOTS) cotton or organic cotton
poplin. This can be referenced to having classic shirts in a catalog made of
Organic/GOTS cotton and peplum/ruffle hem as features for the shirt. Based on
these attribute insights, assortment planners would work closely with
suppliers and designers to curate a collection of such clothing items to
complete the look, including leggings, sports bras, sweatshirts, and sneakers.
They would consider factors such as quality, affordability, and inclusively to
ensure that the assortment caters to a wide range of customers. The images
shows popular prints, innovative fabrics, and style variations within the
classic shirt category. This would enable retailers to offer a diverse
selection of classic shirt options that align with the latest trends.
Additionally, once retailers have an assortment ready for selling, they could
collaborate with fashion influencers who align with the upcoming trend to
create exclusive collections or promote the existing assortment. This would
help to generate excitement among customers and drive high engagement. By
incorporating the recommended color palettes, visual elements, and messaging,
the retailer could create an immersive shopping experience.

Previous Works In this section, we provide a brief overview of existing Multi
Modal Attribute Extraction (MMAE) techniques being used to extract product
attributes from Images and Text. MMAE explained in paper [10] talks about
returning values for attributes which occur in images as well as text and they
do not treat the problem as a labeling problem. They define the problem as
following: Given a product $i$ and a query attribute $a$, extract a
corresponding value $v$ from the evidence provided in terms of textual
description of it ($D_{i}$) and a collection of images ($I_{i}$). For
training, for a set of product items $I$, for each item $i\in I$, its textual
description Di and the images $I_{i}$, and a set $A_{i}$ comprised of
attribute-value pairs $A_{i}=\\{\langle a^{j}_{i},v^{j}_{i}\rangle\\}_{j}$.
The model is composed of three separate modules: (1) an encoding module that
uses modern neural architectures to jointly embed the query, text, and images
into a common latent space, (2) a fusion module that combines these embedded
vectors using an attribute-specific attention mechanism to a single dense
vector, and (3) a similarity-based value decoder which produces the final
value prediction. Another approach for MMAE explained in paper [5], considers
cross-modality comparisons. They leverage pre-trained deep architectures to
predict attributes from text or image data. By applying several refinements to
leverage pre-trained architectures and build single modality models like Text
only modality model, image only modality model for the task of product
attribute prediction. A new modality merging method was proposed to mitigate
modality collapse. For every product, it lets the model assign different
weights to each modality and introduces a principled regularization scheme to
mitigate modality collapse. Paper by [14] talks about Multi modal Joint
Attribute Prediction and Value Extraction for E-commerce Product. They enhance
the semantic representation of the textual product descriptions with a global
gated cross-modality attention module that is anticipated to benefit attribute
prediction tasks with visually grounded semantics. Moreover, for different
values, the model selectively utilizes visual information with a regional-
gated cross-modality attention module to improve the accuracy of value
extraction. Note: As these methods are industry related, hence source code is
not publicly available to reproduce the outcome.

Challenges Despite the potential, leveraging PDF reports consisting text and
images for attribute value extraction remains a difficult problem. We
highlight few challenges faced during designing and executing extraction
framework:

  * •

C1: Text Extraction from PDF: PDF reports can be a combination of multiple
images, overlapping text elements, annotations, metadata and unstructured text
integrated together in no specific PDF format. Extracting text from such
reports can be difficult, challenging and lead to misspelled text and loss of
specific topic-related context. Another issue is missing and noisy attributes.
Text data might not have all the attributes which we are looking for.
Therefore, visual attribute extraction plays an important role.

  * •

C2: Image Extraction from PDF: Images in PDF reports can be embedded,
compressed down to reduce size, in various formats like JPEG, PNG etc.
Extracting images while maintaining the resolution and quality of images
requires specialized handling to accurately preserve the original appearance.
Also, images could bring multi-labeled attributes which can confuse the model
but can be mitigated by merging certain attribute values to help with model
inferences.

  * •

C3: Extracting Product Attributes: Product tags extracted from text/images
needs to be carefully mined to match product attributes. The attributes differ
based on the category of products we are referring to and can have multi-
labeled attributes. For example, women’s tops will have sleeve related
attribute whereas women’s trousers will have type of fit attribute and sleeve
attribute will be irrelevant.

  * •

C4: Mapping Product Attributes to Product Catalog: E-commerce catalog has
specific products and attributes mapped to them. On-boarding new attributes
based on PDF reports, requires new attribute creation/refactoring existing
attributes.

Informal Problem 2. Can we develop unsupervised models that require limited
human annotation? Additionally, can we develop models that can extract
explainable visual attributes, unlike black-box methods that are difficult to
debug?

Mitigating Challenges: Current multi-modal attribute extraction solutions [10,
5] are inadequate in the e-commerce field when it comes to handling challenges
C2 and C4. Conversely, text extraction solutions that successfully extract
attribute values are primarily text-oriented [6, 13, 2, 12] and cannot be
easily applied to extracting attributes from images. In this work, we address
the central question: how can we perform multi-modal product attribute
extraction from upcoming trend PDF reports? The detail description is given in
section III to handle each challenge. Our proposed method PAE works on
extracting upcoming trends from PDF reports generated by the trend forecasting
company. This capability provides an insight into upcoming marketing trends
and customer preferences. By using trend forecasting reports, catalog can be
refined with new classes of products having trending attributes based on
external reports, to propel value across the apparel space by accurately
indicating attribute trends in the market and increasing customer
satisfaction. The contributions of our paper are as follows:

  * •

Novel Problem Formulation: We propose the end-to-end model of jointly
extracting the trending product attributes and hashtags from PDF files
consisting of text and image data and mapping it back with the product catalog
for the final product attributes values. An example of end to end execution of
product attribute extraction and mapping is shown in the figure 2. Due to
Walmart Privacy Requirements, models and datasets are not open to public. We
have elaborated the details of each model, and readers can use LLM model of
their own choice.

  * •

Flexible Framework: We develop a general framework PAE for extracting text and
images from PDF files and then generating product attributes. All the
components are easily modified to enhance the capability or to use the
framework partially for other applications. The extraction engine can be used
to extract attributes for different categories of products like Electronics,
Home decor etc.

  * •

Experiments: We performed extensive experiments in real-life datasets to
demonstrate PAE ’s efficacy. It successfully discovers attribute values from
text and image data with a high F1-score of 96.8%, outperforming state-of-the-
art models. This proves its ability to produce stable and promising results.

The remainder of the paper is organized as follows: The problem formulation is
given in Section II. In Section III we describe our proposed method PAE in
details with examples. Finally, we show the experimental results in section IV
and section V concludes the paper.

Figure 3: Overview of the proposed Product Attribute Extraction Engine. 1) PDF
Data Extraction: Text and Images. 2) Text and Image Attribute Extraction: By
using Large Language Models(LLM) models we extract various attributes like
color, age etc. 3) Catalog Matching: We create attribute embeddings for
extracted attributes and existing catalog attributes. Based on high cosine
similarity, we send signal to future assortment planning teams.

##  II Problem Definition

The upcoming trend information in PDF files usually looks like figure 1. The
text describes the upcoming trends and style types. The images along with it
shows how the style will look on different models. We consider each page of
PDF file as one product type and that can be Woven Tops, Knitwear etc. We
formally define the problem as follows.

Problem definition: With the following information:  Given (a) a PDF file with
multiple pages $[1,2,3\dots,N]$ consist of text $T_{1}^{N}$ and image
$I_{1}^{N}$ data (b) LLM prompt $P$ with target attributes $attr\in$ [color,
sleeve style, product type, material, features, categories, age and neck ]
Find the value $vals$ for the target attribute $attr$ related for each page.

Figure 1 displayed a few such attribute values from Text and Image in the PDF
file. Specifically, considering the target attribute Material shown in figure
1 \- C, our objective is to extract the attribute value Cotton. If the target
attribute is Neck in figure 1 \- B from image, the objective is to extract the
attribute value V-Neck. Similarly, target attribute Feature for figure 1 \-
(A, B, C, D), our objective is to extract the attribute value Softly Draped,
Bow belt, Peplum and Folded respectively.

##  III Product Attribute Extraction

In this work, we tackle the attribute value extraction as pair task, i.e.,
extracting the attribute values from image and text together. The input of the
task is a “textual information $T$, set of images $I_{1},I_{2}\dots I_{N}$
pair per PDF page, and the output is the product attributes values. Our
framework is presented in figure 3. In fact, through extensive experiments
(see Section IV), we show that our proposed method is not only intuitive, but
achieves state-of-the-art performance compared to previously proposed methods
in literature. Due to Walmart Privacy Requirements, models and datasets are
not open to public. We have elaborated the details of each model, and you can
use LLM model of your choice.

###  III-A Text Extraction from PDF

Text extraction from PDF is an important process that involves the conversion
of data contained in PDF files into an editable and searchable format. This
procedure is crucial for activities like data analysis, content re-purposing,
and detecting trends from public reports. However, it can pose certain
challenges. The layout complexity of a PDF document can make the extraction
process difficult. For instance, the presence of multiple columns, images,
tables, and footnotes can complicate the extraction of pure text. Another
challenge is the use of non-standard or custom fonts in PDFs, which can lead
to inaccurate extraction results. Moreover, the presence of ’noise’ such as
headers, footers, HTML tags and page numbers can also interfere with the
extraction process.

Figure 4: Text Extraction via Image Annotator

There are numerous tools available for text extraction from PDF files.
Searching for text extraction from PDF on Google yields a plethora of results
featuring various tools or pages suggesting such tools as pdfMiner [11],
pdfquery[4] etc. However, figure 4 represents the process we used to extract
the text from pdf files. First, we split the PDF files into PIL (Python
Imaging Library) images using ”convert from path” function from the pdf2image
[1]. Internally, the function uses the pdfinfo command-line tool to extract
metadata from the PDF file, such as the number of pages. It then uses the
pdftocairo command-line tool to convert each page of the PDF into an image.
Second, we convert the images to grayscale and perform morphological
transformations on each page by applying a morphological gradient operator to
enhance and isolate text regions. Finally, we use Image Annotator [7] consists
of Optical Character Recognition (OCR) capabilities for text extraction. Once
the text is extracted, we use the spell Corrector like language-tool to fix
any misinterpreted text from OCR. The Text extracted from PDF report about
product type ”Slouchy V-Neck” is given below:  The Slouchy V-Neck Why is it
key? Slouchy, over- sized and boxy silhouettes continue to drive casual V-neck
styles, in line with the growing desire for a cosy, comfortable wardrobe that
reassuringly envelops the wearer for A/W. A recognisable knitwear shape ripe
for a youthful update, the #countrycalling trend we’re tracking is reworking
classic patterns in larger scales. #Supersizedintarsia argyle motifs are a
bold and contemporary example. Design details: use 5gg yarn, ensuring a deep
V-neckline and raglan sleeves to heighten the slouchy look. Knit in
sustainable brushed super-kid mohair yarn for a cosy, soft finish, using a
palette of earthy greens and grey tones.

###  III-B Image Extraction from PDF

PDF files can contain images in various formats such as JPEG, PNG, or TIFF.
Extracting images from different formats may require multiple techniques. The
second challenge could be different types of images in the pdf files,
including scanned documents, vector graphics, or embedded images. Third,
extracting images from large PDF files efficiently and in a timely manner can
be a challenge, especially when dealing with limited system resources. To
tackle the above-mentioned challenges, we exploit pure-python PDF library [3],
as a standalone library for directly extracting image objects from PDF files.
With pure-python PDF library, we identify the pages with images and extract
them as raw byte strings. Then, using Pillow, the extracted images are
processed and saved in jpg formats. Figure 5 shows extracted images from
files.

Figure 5: Example of extracted images from PDF files

###  III-C Attribute Extraction from Text

PDF reports consist of specific products or product categories, providing
details on their design, features, materials, colors, and styles. These
reports provide information about product innovations, market demand, and
consumer preferences in upcoming months or years. Here, we extract 8 product
attributes namely [Color, Sleeve Style, Product Type, Material, Features,
Categories, Age, Neck]. We also extract hashtags to discover and explore
content related to a specific topic or theme in catalog.

We are utilising the LLM model for extracting the attributes. Below is a
sample prompt:

Generate me color, sleeve style, product type, material, features, categories,
age and neck attributes of below text:
a text $T_{i}$

Example
Generate me color, sleeve style, product type, material, features, categories,
age and neck attributes from the following text:
”The Slouchy V-Neck Why is it key? Slouchy, oversized and boxy silhouettes
continue to drive casual V-neck styles, in line with the growing desire for a
cosy, comfortable wardrobe that reassuringly envelops the wearer for A/W. A
recognisable knitwear shape ripe for a youthful update, the #countrycalling
trend we’re tracking is reworking classic patterns in larger scales.
#Supersizedintarsia argyle motifs are a bold and contemporary example.
Design details: use 5gg yarn, ensuring a deep V-neckline and raglan sleeves to
heighten the slouchy look. Knit in sustainable brushed super-kid mohair yarn
for a cosy, soft finish, using a palette of earthy greens and grey tones.”

The output is then processed in dictionary type as follow:  Output of above
text example
Color: Earthy Greens And Grey Tones
Sleeve Style: Raglan Sleeves
Product Type: V-Neck Sweater
Material: Sustainable Brushed Super-Kid Mohair Yarn
Features: Cosy, Soft Finish
Categories: Casual, Knitwear
Age: Youthful
Neck: Deep V-Neckline

###  III-D Attribute Extraction from Images

The extraction of detailed image attributes from fashion images has a wide
range of uses in the field of e-commerce. The recognition of visual image
attributes is vital for understanding fashion, improving catalogs, enhancing
visual searches, and providing recommendations. In fashion images, the
dimensionality can be higher due to the complexity and diversity of fashion
items. For instance, a single piece of clothing can have multiple attributes
for color, fabric type, style, design details, size, brand, and others. Hence,
image attribute extraction has become more complex than text. However, these
attributes can be extracted using various computer vision techniques, such as
image segmentation, object detection, pattern recognition and deep learning
algorithms. In this work, we explore the vision based LLM model. Each
extracted image as shown in figure 6 is converted to base64 encoding. Base64
encoding is a method of converting binary data, such as an image, into ASCII
text format. This is required as current LLM models takes text format as
input. The ASCII text format example as follow:

Figure 6: Image Example

ASCII text format
’iVBORw0KGgoAAAANSUhEUgAAAU4AAA
GwCAIAAABJqRtXAAAACXBIWXMAAA7EAAAOxAGV
Kw4bAAEV4klEQVR4nOy9d0BUV9f2HRXEjgU
QBcWCscXeNfbebnuLHbtYsPfeY1cEG2IBbKjYF
…’

Next, we use this encoded string along with LLM prompt to generate the product
attributes as follow:  Example
text: ”Generate a list format color, sleeve style, product type, material
attributes from the the below image. Also give me features, categories, age
and neck attributes from below image.”
”inlineData”: {
”mimeType”: ”image/png”,
”data”: ’iVBORw0KGgoAAAANSUhEUgAAAU4AAA
GwCAIAAABJqRtXAAAACXBIWXMAAA7EAAAOxAGV
Kw4bAAEV4klEQVR4nOy9d0BUV9f2HRXEjgU
QBcWCscXeNfbebnuLHbtYsPfeY1cEG2IBbKjYF
…’ }

The output is then processed in dictionary type as follow:  Output of above
image example
Color: Multicolor,
Sleeve Style: Long Sleeve
Product Type: Pullover
Material: Wool Blend
Features: V-Neck, Drop Shoulder
Categories: Women’s Fashion
Age: Adult
Neck: V-Neck

Another common issue that arises is the presence of noisy and missing labels.
It is a challenging task to accurately label and annotate all the relevant
information for every page in the PDF. Despite employing various automated and
manual annotation processes, it is nearly impossible to obtain perfectly
labeled structured data. To address this, we employ image pre-processing or
data cleaning techniques to eliminate duplicate, noisy, and invalid images
before proceeding with attribute extraction. Once we extract attributes from
text and images on each page, we aggregate them per page for our further
analysis.

###  III-E Hashtag Detection in Text

Hashtags are words or phrases preceded by the pound sign (#) and are commonly
used on social media platforms to categorize and group similar content.
Detecting hashtags in text is crucial for various applications such as topic
modeling, sentiment analysis, and product recommendation. The process of
hashtag detection involves analyzing the text and identifying words or phrases
that are preceded by the pound sign, while considering factors such as word
boundaries and punctuation marks. The extracted hashtags can then be used to
gain insights into trending topics, user interests, or to enhance search and
recommendation systems. In our work, we use the regular expression
${}^{\prime}\\#w+^{\prime}$ to detect the hashtags.  Hashtags extracted from
above example text
#countrycalling
#Supersizedintarsia

###  III-F Product attribute Matching

The purpose of product attribute matching is to ensure that extracted
attributes meet specific criteria or requirements. Based on the identified
attributes, retailers plan their future product assortments. This includes
selecting or designing clothing items, accessories, and other fashion products
that reflect the upcoming trends. One of the challenge of product attribute
matching is multiple variants of representation for the same value of one
attribute. For example, ”vneck” and ”V-Neck” is consolidated into ”V-Neck” as
neck product attribute. Figure 7 shows our framework to match the predicted
attributes to the existing catalog attributes. We exploit pre-trained bert
uncased model. BERT is designed to pre-train deep bidirectional
representations from unlabeled text by jointly conditioning on both left and
right context in all layers. We create representations or embeddings for
predicted attributes and existing product attributes. Finally, we use cosine
similarity to match the similar product attributes from the catalog.

Figure 7: Product attribute Matching Framework

Summary: As shown in figure 2, our proposed approach relies on four successive
steps. First, the text and image extraction step that extracts all the text
(paragraphs) and relevant images from the given PDF files. Second, attribute
extraction step, that uses LLM models to extract relevant attributes from the
images and text. Third, merging step, we consolidate attributes into each
category and keep unique values for each attribute. Finally, catalog matching
step, helps retailers to find a product that matches with existing inventory
and plan for future assortment.

##  IV Experiments

Although our work is mostly related to retail business, we will compare the
performance of our PAE’s sub-parts with different baselines on real-life
datasets. We evaluate our approach on 14 upcoming trend reports. In
particular, we want to answer the following questions:

  * •

(Q1) How accurate is our proposed method PAE when compared to other baselines?

  * •

(Q2) How sensitive is PAE w.r.t different parameters?

  * •

(Q3) How time consuming is PAE?

###  IV-A Data-set description

We provide the datasets used for evaluation in Table I. These are trend
reports used for 2023 assortment planning. Assuming the attribute value is
applicable to each pdf page, if the attribute value information can not be
observed from the given text description and images, we will assign “Not
Mentioned” as the corresponding value.

Dataset | $\\#P$ | $\\#T$ | $\\#I$ | $\\#H$ | $GT$
---|---|---|---|---|---
Boys Apparel | $7$ | $11$ | $32$ | $0$ | $Y$
Women’s Cut Sew | $7$ | $30$ | $24$ | $5$ | $Y$
Women’s Woven Tops | $6$ | $28$ | $24$ | $6$ | $Y$
Country Life Boys | $12$ | $12$ | $66$ | $3$ | $Y$
Knitwear Jersey | $10$ | $13$ | $54$ | $8$ | $Y$
Modern Occasion | $10$ | $16$ | $60$ | $10$ | $Y$
Jackets Outerwear | $7$ | $9$ | $9$ | $7$ | $Y$
Woven Tops | $7$ | $10$ | $9$ | $6$ | $Y$
Knitwear Core | $6$ | $11$ | $31$ | $4$ | $Y$
Knitwear Fashion | $12$ | $21$ | $70$ | $7$ | $Y$
Woven Tops Core | $6$ | $13$ | $35$ | $0$ | $Y$
Woven Tops Fashion | $13$ | $24$ | $74$ | $4$ | $Y$
TABLE I: Datasets used in the experiments. For each PDF file, we extracted all
the pages. $\\#P$ represents number of pages, $\\#T$ represents number of text
blocks, $\\#I$ represents number of total images present in pdf file. Here
$\\#H$ represents hashtags available in pdf file and $GT$ is ground truth
attributes are available for the pdf file.

###  IV-B Evaluation Measures

We use Accuracy, True Positive Rate (Recall) and F1 score as the evaluation
metrics. We compute Accuracy (denoted as $P$) as percentage of correct value
generated by our framework; True positive rate (denoted as $TPR$) as
percentage of ground truth value retrieved by our framework; F1 score (denoted
as $F1$) as harmonic mean of Precision and Recall.

###  IV-C Baselines

To evaluate our proposed framework, we choose the following models as
baselines for text: topic rank [2] and sOpenTag [12]. Our attribute value
extraction task for images is highly related to the visual question answering
tasks. Thus, we used two baselines, Vilt [8], and BLIP [9] for visual
attribute value extraction.

###  IV-D Accuracy of PAE

For all datasets we compute F1-score (%) for text and images. The results for
qualitative measure for data is shown in Table II. We observed that F1-score
(Image) is perfect for all the dataset as images provide clear visual
attributes for future trends. However, text data has missing attributes and
PAE is able to extract average $92.5\%$ of attributes from the all the PDF
files.

Dataset | F1-score (Text) | F1-score (Image)
---|---|---
Boys Apparel | $94.3\%$ | $100\%$
Women’s Cut Sew | $88.6\%$ | $100\%$
Women’s Woven Tops | $100\%$ | $100\%$
Country Life Boys | $89.4\%$ | $100\%$
Knitwear Jersey | $82.4\%$ | $100\%$
Modern Occasion | $92.8\%$ | $100\%$
Jackets Outerwear | $89.4\%$ | $100\%$
Woven Tops | $86.6\%$ | $100\%$
Knitwear Core | $92.5\%$ | $100\%$
Knitwear Fashion | $97.8\%$ | $100\%$
Woven Tops Core | $96.8\%$ | $100\%$
Woven Tops Fashion | $98.9\%$ | $100\%$
TABLE II: Test accuracy for multiple datasets for PAE

Further we compare the performance of PAE with the aforementioned baseline
values for text and images separately. We use ”Woven Tops Core” dataset with 8
pages. We evaluate text extraction on 6 pages out of 8 with 7 attributes
namely Sleeve Style, Features, Product Type, Material, Neck Style, Product
Categories and Color. From these 7 attributes, we have total 33 attribute
values and 9 not mentioned values. Table III shows the performance of our
method compared to baseline methods. Topic rank and sOpenTag performed very
well on ’Woven Tops Core’ dataset. However, PAE outperforms baselines in-terms
of F1 score and accuracy. We also note that our proposed method is $2.7\times$
faster as compared to Topic rank and sOpenTag.

Dataset | PAE |  Topic Rank [2] | sOpenTag[12]
---|---|---|---
Precision | $100\%$ | $93.3\%$ | $61.4\%$
True Positive Rate | $93.9\%$ | $42.4\%$ | $73.4\%$
Accuracy | $95.3\%$ | $54.7\%$ | $86.2\%$
F1-Score | $96.8\%$ | $59.5\%$ | $66.7\%$
TABLE III: Text Attribute extraction accuracy for ’Woven Tops Core’ datasets
for PAE and state-of-art-methods.

We evaluate image attribute extraction on all 8 attributes namely Sleeve
Style, Product Type, Material , Neck, Categories, Age, Features and Color for
all 8 pages with image information. From these 8 attributes, we have total 64
visual attribute values. Table IV shows that PAE outperforms the baseline
methods. Both the baselines were limited to pass single attribute in prompt,
therefore results in consuming more time in producing the final attribute
values per page. This answer our question Q1.

Attributes | PAE |  Vilt [8] | BLIP[9]
---|---|---|---
Color | $100\%$ | $87.5\%$ | $87.5\%$
Sleeve Style | $100\%$ | $00.0\%$ | $50.0\%$
Product Type | $100\%$ | $62.5\%$ | $62.5\%$
Material | $100\%$ | $75.0\%$ | $75.0\%$
Features | $100\%$ | $50.0\%$ | $25.0\%$
Categories | $100\%$ | $75.0\%$ | $100\%$
Age Group | $100\%$ | $75.0\%$ | $00.0\%$
Neck | $100\%$ | $12.5\%$ | $62.7\%$
TABLE IV: Accuracy per attribute for ’Woven Tops Core’ dataset for PAE and
state-of-art-methods on images.

###  IV-E Parameter of sensitivity for PAE

####  IV-E1 Sensitivity to LLM Prompt for Text data

Large Language Models (LLMs) have the ability to learn new tasks on the fly,
without requiring any explicit training or parameter updates. This mode of
using LLMs is called in-context learning. It relies on providing the model
with a suitable input prompt that contains instructions and/or examples of the
desired task. Therefore, we evaluate our proposed method PAE for multiple
prompts for text and image attribute extraction. Here, we present prompt
analysis for attribute extraction from text data only. We keep the temperature
parameter constant for this experiment.

  * •

Prompt 1: ”Give me all clothing characteristics of a product from the
following text:”

  * •

Prompt 2: ”Give me color, sleeve style, product type, material, cloth
features, categories, and neck attributes from the following text:”

  * •

Prompt 3: ”I want you to act as a product attribute extractor in retail
space.Given the unstructured text data, you need to find different product
attributes in the text. For example: For Input as ‘Long contrast fabric Sleeve
red cotton adult polo shirts for men with contemporary design element’, the
attribute extractor will return color attribute is red, sleeve attribute is
Long, style sleeve attribute is contrast fabric, product type attribute is
polo shirts, material attribute is cotton, feature attribute is
contemporary,categories is polo shirts, gender attribute is men and neck
attribute is NA. Give me attributes like color, sleeve style, product type,
material, features, categories, and neck attributes from the following text: ”

Dataset | Prompt 1 | Prompt 2 | Prompt 3
---|---|---|---
Precision | $83.3\%$ | $100\%$ | $90.4\%$
True Positive Rate | $15.5\%$ | $93.9\%$ | $57.6\%$
Accuracy | $30.9\%$ | $95.3\%$ | $61.9\%$
F1-Score | $25.6\%$ | $96.8\%$ | $70.3\%$
TABLE V: Sensitivity to LLM Prompt for ’Woven Tops Core’ dataset. Figure 8:
Running time for each PDF report.

The table V shows that Prompt 2 is more effective and efficient way to extract
given attributes from the text data. Too vague (Prompt 1) and too much (Prompt
3) information/context confuses the LLM model and therefore performance
degrades.

####  IV-E2 Sensitivity to LLM Parameters

In this study, we evaluate the performance of temperature parameter of LLM
models. Temperature is a parameter in large language models (LLMs) that
controls the randomness of the model’s responses, ranging from 0 to 1. A
higher temperature means more creative and diverse output, while a lower
temperature means more predictable output. As we need predictable output, we
kept the temperature parameter below 0.5. The table VI shows that at
temperature = $0.2$, method performance is high. Therefore, we chose to keep
$0.2$ as parameter value.

Dataset | $0.05$ | $0.1$ | $0.2$ | $0.4$
---|---|---|---|---
Precision | $85.7\%$ | $93.7\%$ | $100\%$ | $100\%$
True Positive | $72.2\%$ | $90.9\%$ | $93.9\%$ | $90.9\%$
Accuracy | $78.5\%$ | $88.1\%$ | $95.3\%$ | $92.8\%$
F1-Score | $84.2\%$ | $92.3\%$ | $96.8\%$ | $95.2\%$
TABLE VI: Sensitivity to LLM temperature parameters for attribute extraction
from text data in ’Woven Tops Core’ dataset.

In summary, as expected, PAE is sensitive to both LLM prompt and temperature
parameter. This answer our question Q2.

###  IV-F CPU Time Analysis

In this work, we present two experiment results. First, we provide CPU time
for each dataset in figure 8. We observe that CPU time is directly
proportional to number of images in the PDF files. ”Women Tops Fashion” PDF
file has 74 images and took around $350$ seconds to get attributes for all
$13$ pages.

Figure 9: Running time for synthetic PDF report.

Second, we created synthetic PDF files with each consisting of 500-1000 words
and 4-6 images per page. We created PDF files with
$[10,50,100,500,1000,5000,10000]$ pages. The figure 8 shows that CPU time is
linear w.r.t size of the PDF file. This answer our question Q3.

##  V CONCLUSIONS AND FUTURE WORK

In this work, we have described an attribute extraction framework namely PAE
for applications in the domain of inventory/E-commerce business. The goal is
given a PDF file consisting of upcoming trends in the form of text and images,
our framework correctly extracts the defined attributes in order to plan the
future assortment and associate these predicted attributes to the current
product in a catalog for better planning. The performance of the proposed
method is assessed via experiments on real-world datasets. We summarize our
contribution as:

  * •

The proposed framework effectively identifies the attributes of from PDF files
to achieve an assortment planning task. To further enhance the capability, we
tailored our proposed framework towards flexibility where extraction of data
and attributes can be easily enhanced and modified for domain-specific
applications.

  * •

Through experimental evaluation on multiple datasets, we show that PAE
provides accurate attributes and is significantly faster in terms of CPU run
time.

There is still room for improving our methods. One direction is to explore LLM
models which can take sets of images & text and provide consolidate
attributes. Another direction is to further improve the product matching
system that consists of product images so that our method can be more suitable
for customers during the search for different products on e-commerce websites.

##  VI Acknowledgements

Research was supported by the Walmart. Any opinions, findings, and conclusions
or recommendations expressed in this material are those of the author(s) and
do not necessarily reflect the views of the funding parties.

## References

  * [1] Edouard Belval.  pdf2image.  https://pypi.org/project/pdf2image/, 2017.
  * [2] Adrien Bougouin, Florian Boudin, and Béatrice Daille.  Topicrank: Graph-based topic ranking for keyphrase extraction.  In International joint conference on natural language processing (IJCNLP), pages 543–551, 2013.
  * [3] claird.  Pypdf4.  https://pypi.org/project/PyPDF4/, 2018.
  * [4] Jack Cushman.  Pdfquery.  https://github.com/jcushman/pdfquery/tree/master, 2013.
  * [5] Aloïs De la Comble, Anuvabh Dutt, Pablo Montalvo, and Aghiles Salah.  Multi-modal attribute extraction for e-commerce.  arXiv preprint arXiv:2203.03441, 2022.
  * [6] Pushpendu Ghosh, Nancy Wang, and Promod Yenigalla.  D-extract: Extracting dimensional attributes from product images.  In WACV 2023, 2023.
  * [7] Google.  Google cloud vision api.  https://cloud.google.com/python/docs/reference/vision/latest.
  * [8] Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision-and-language transformer without convolution or region supervision.  In International conference on machine learning, pages 5583–5594. PMLR, 2021.
  * [9] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International conference on machine learning, pages 12888–12900. PMLR, 2022.
  * [10] Robert L Logan IV, Samuel Humeau, and Sameer Singh.  Multimodal attribute extraction.  arXiv preprint arXiv:1711.11118, 2017.
  * [11] PYusuke Shinyama.  pdfminer.  https://www.unixuser.org/~euske/python/pdfminer/, 2004.
  * [12] Huimin Xu, Wenting Wang, Xinnian Mao, Xinyu Jiang, and Man Lan.  Scaling up open tagging from tens to thousands: Comprehension empowered attribute value extraction from product title.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5214–5223, 2019.
  * [13] Guineng Zheng, Subhabrata Mukherjee, Xin Luna Dong, and Feifei Li.  Opentag: Open attribute value extraction from product profiles.  In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1049–1058, 2018.
  * [14] Tiangang Zhu, Yue Wang, Haoran Li, Youzheng Wu, Xiaodong He, and Bowen Zhou.  Multimodal joint attribute prediction and value extraction for e-commerce product.  arXiv preprint arXiv:2009.07162, 2020.


# Bootstrapping Informative Graph Augmentation via A Meta Learning Approach

Hang Gao1,211footnotemark: 1 Jiangmeng Li1,2111Contributed equally to this
work, in no particular order. Wenwen Qiang1,2222Corresponding author Lingyu
Si1,2 Fuchun Sun3
Changwen Zheng2 1University of Chinese Academy of Sciences
2Institute of Software Chinese Academy of Sciences
3Tsinghua University
{gaohang, jiangmeng2019, wenwen2018, lingyu<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recent works explore learning graph representations in a self-supervised
manner. In graph contrastive learning, benchmark methods apply various graph
augmentation approaches. However, most of the augmentation methods are non-
learnable, which causes the issue of generating unbeneficial augmented graphs.
Such augmentation may degenerate the representation ability of graph
contrastive learning methods. Therefore, we motivate our method to generate
augmented graph with a learnable graph augmenter, called MEta Graph
Augmentation (MEGA). We then clarify that a ”good” graph augmentation must
have uniformity at the instance-level and informativeness at the feature-
level. To this end, we propose a novel approach to learning a graph augmenter
that can generate an augmentation with uniformity and informativeness. The
objective of the graph augmenter is to promote our feature extraction network
to learn a more discriminative feature representation, which motivates us to
propose a meta-learning paradigm. Empirically, the experiments across multiple
benchmark datasets demonstrate that MEGA outperforms the state-of-the-art
methods in graph self-supervised learning tasks. Further experimental studies
prove the effectiveness of different terms of MEGA. Our codes are available at
https://github.com/hang53/MEGA.

##  1 Introduction

Recently, there has been a surge of interest in learning a graph
representation via self-supervised Graph Neural Network (GNN) approaches.
GNNs, inheriting the powerful representation capability of neural networks,
emerged as benchmark approaches over many graph representation learning tasks.
Early works mostly require task-dependent labels to learn a graph
representation. However, annotating graphs is a rather challenging task
compared to labeling common modalities of data, especially in specialized
domains. Therefore, recent research efforts are dedicated to developing self-
supervised graph representation learning methods, which can eliminate the
dependency of the labels Hu et al. (2020a).

Graph contrastive learning (GCL), one of the most popular self-supervised
methods in graph representation learning, is proposed based on GNNs and
contrastive learning. Under the learning paradigm of contrastive learning, GCL
generates augmented graphs by adopting graph augmentation Hassani and
Khasahmadi (2020). After graph encoding, the augmented and original features
of the same graph are treated as positives, and the features of different
graphs are treated as negatives. The object of GCL is to learn a good graph
representation by pulling the positives close and pushing the negatives apart.
However, most of the graph augmentation approaches are non-learnable, which
causes two issues: 1) the augmentation is excessively weak, e.g., the
augmented graph is indistinguishable from the original graph, and the
contrastive learning model can hardly mine consistent knowledge from them; 2)
the augmentation introduces overmuch noise, and the augmented graph is even
more similar to other graphs. The mentioned issues weaken GCL’s ability to
learn a discriminative representation. Therefore, we motivate our method to
directly learn a graph augmentation, which can assist GCL in generating a good
graph representation.

We aim to learn a ”good” graph representation that can have impressive
performance on downstream tasks, but what is an exact ”good” representation?
From Grill et al. (2020) Ermolov et al. (2021), we notice that, at the
instance-level, a good representation naturally has uniformity, e.g., features
of different samples are scattered throughout the hidden space instead of
collapsing to a point. However, such constraint does not consider the
representation’s collapse at the feature-level. For instance, the learned
representation has 256 dimensions, but most of them have few differences,
which implies that much information learned by the representation is redundant
Zbontar et al. (2021). Such redundant information may lead to limited
informativeness of the representation and degenerate the representation to
model truly discriminative information. Therefore, we motivate our method to
learn a ”good” representation with uniformity at the instance-level and
informativeness at the feature-level.

Figure 1: An illustration example of the training process of MEGA. The figure
shows the features in hidden space during training. Red points depicts the
positive features and blue points depicts the negative features at the
instance-level. The gradation of color denotes the informativeness at the
feature-level.

To this end, we propose MEta Graph Augmentation (MEGA) to guide the encoder to
learn a discriminative and informative graph representation. For training the
encoder, we follow the common setting of contrastive learning Chen et al.
(2020). The well self-supervised contrastive learning approach leads the
features to be scattered in the hidden space. However, in practice, the
sufficient self-supervision of the contrastive learning approach demands hard
(informative at the instance-level) features of positive and negative samples,
e.g., positive features that are relatively far apart and negative features
that are relatively close in the hidden space. For instance, Chen et al.
(2020) leverages large batch and memory bank to include more hard features,
Chuang et al. (2020) explore to emphasize hard features in training. At the
instance-level, our motivation is to straightforwardly generate hard graph
features by a learnable graph augmenter (LGA), which uses the encoder’s
performance in one iteration to generate hard features for the next iteration
by updating the graph augmentation. Note that the objective of LGA is not to
promote convergence of contrastive loss. On the contrary, we expect LGA to
generate an augmented graph that can increase the difficulty of the self-
supervision problem (i.e., contrasting). Contrastive learning aims to put the
original and augmented features of a graph together and push the features of
different graphs away, and LGA aims to degenerate such a process. Therefore,
the LGA augmented graph feature must be hard for contrastive learning. To
ensure the informativeness of the learned representation at the feature-level,
we propose to train LGA to augment the graph so that it can improve the
encoder to generate a representation with informativeness at the feature-
level. As shown in Figure 1, LGA is like a teacher that shows different hard
and informative examples (augmented graphs) to the encoder, and the
contrastive loss leads the encoder to learn discriminative knowledge from
them.

The reason why we take a meta-learning approach to update LGA is as follows:
the learning paradigm of meta-learning ensures that the optimization objective
of LGA is improving the encoder to learn representations with uniformity at
the instance-level and informativeness at the feature-level from graphs.
However, a regular learning paradigm, e.g., directly optimizing LGA by the
loss of measuring the uniformity and informativeness of features in hidden
space, can only ensure that the features learned from the augmented graph are
modified. However, the features learned from the original graph could be
collapsed or non-informative. Concretely, the meta-learning paradigm ensures
that the encoder learns the knowledge to generate good representations with
uniformity at the instance-level and informativeness at the feature-level.

Contributions. The takeaways of this paper are as follows:

  * •

We propose a learnable approach to generate informative graph augmentation,
called meta graph augmentation, which boosts the performance of graph
contrastive learning.

  * •

We propose an auxiliary meta-learning approach to train the learnable graph
augmenter, which guides the encoder to learn a representation with uniformity
at the instance-level and informativeness at the feature-level.

  * •

We conduct experiments to compare our method with state-of-the-art graph self-
supervised learning approaches on benchmark datasets, and the results prove
the superiority of our method.

Figure 2: MEGA’s architecture. MEGA uses LGA to generate augmented graph,
which and the original graph are encoded together. In one iteration, the
encoder and projection head are trained by back-propagating
$\mathcal{L}_{contrast}$, and in the next iteration, the LGA is trained by
performing the second-derivative technique on $\mathcal{L}_{MEGA}$. The
encoder is trained until convergence.

##  2 Related Works

In this section, we review some representative works on graph neural networks,
graph contrastive learning, and meta-learning, which are related to this
article.

#### Graph Neural Networks (GNN).

GNN can learn the low-dimensional representations of graphs by aggregating
neighborhood information. These representations can then be applied to various
kinds of downstream tasks. Like other neural network structures, GNNs
developed many variants. Graph Convolution Networks (GCNs) Kipf and Welling
(2016), as an extension of the convolutional neural network on graph-
structured data, use convolution operation to transform and aggregate features
from a node’s graph neighborhood. Xu et al. (2018) shows that GNNs are at most
as powerful as the Weisfeiler-Lehman test in distinguishing graph structures.
Based on this idea, Xu et al. (2018) proposed Graph Isomorphism Networks
(GINs). Graph Attention Networks (GATs) Veličković et al. (2017) introduces
attention mechanisms into graph learning.

#### Contrastive learning.

Contrastive Learning is a kind of self-supervised learning approach that
measures the loss in latent space by contrasting features in hidden space. CMC
Tian et al. (2020) uses multi-view data to acquire features for contrasting.
In computer vision, many works based on contrastive learning have achieved
outstanding results in different kinds of tasks Chen et al. (2020) Zbontar et
al. (2021). As in graph learning, contrastive learning also has many
applications. For instance, DGI Veličković et al. (2018) learns node
representations through contrasting node and graph embeddings. Hassani and
Khasahmadi (2020) learns node-level and graph-level representations by
contrasting the different structures of a graph.

#### Meta learning.

The objective of meta-learning is to learn the learning algorithm
automatically. Early works Schmidhuber (2014) aim to guide the model (e.g.,
neural network) to learn prior knowledge about how to learn new knowledge, so
that the model can efficiently study new information. Recently, researchers
explored using meta-learning to find optimal hyper-parameters and
appropriately initialize a neural network for few-shot learning Finn et al.
(2017).

##  3 Methods

In this section, we introduce the proposed MEta Graph Augmentation (MEGA). The
architecture of MEGA is depicted in Figure 2. MEGA proposes to learn
informative graph augmentation by a meta-learning approach. Guided by the
augmented graph, the GNN encoder can mine hidden discriminative knowledge from
the original graph.

###  3.1 Preliminary

We recap necessary preliminary concepts and notations for further exposition.
In this paper, we consider attributed graphs $\bm{G}=(\bm{V},\bm{E})$ where
$\bm{V}$ is a node set and $\bm{E}$ is the corresponding edge set. For
$\bm{G}$, $\\{\bm{X_{v}}\in\mathbb{R}^{\bm{V}}|\bm{v}\in\bm{V}\\}$ denotes the
node attributes.

#### Learning graph representations.

Given a graph dataset $\mathcal{G}$ including $\bm{G}_{i}$, where
$i\in\llbracket{1,N}\rrbracket$. Our objective is to learn an encoder
$f(\cdot):\mathcal{G}\to\mathbb{R}^{\bm{D}}$, where $f(\bm{G}_{i})$ is a
representation that contains discriminative information of $\bm{G}_{i}$ and
can be further used in downstream task. We assume $\bm{G}_{i}$ as a random
variable that is sampled i.i.d from distribution
$\mathcal{P}\left(\mathcal{G}\right)$ defined over $\mathcal{G}$. To learn
such discriminative representation $f(\bm{G}_{i})$, we adopt GNN as the
encoder and then perform self-supervised contrastive learning in hidden space.

#### Graph Neural Networks.

In this paper, we focus on using GNN, message passing GNN in particular, as
the encoder $f(\cdot)$. For graph $\bm{G}_{i}=(\bm{V}_{i},\bm{E}_{i})$, we
denote $\bm{H}_{\bm{v}}$ as the representation vector, for each node
$\bm{v}\in\bm{V}_{i}$. The $k$-th layer GNN can be formulated as:

$\displaystyle{\bm{H}^{(k+1)}_{\bm{v}}}=combine^{(k)}\Big{(}\bm{H}^{k}_{\bm{v}},aggregate^{(k)}(\bm{H}^{k}_{\bm{u}}$
$\displaystyle,\forall\bm{u}\in\mathcal{N}(\bm{v}))\Big{)},$ (1)

where $\mathcal{N}(\bm{v})$ denotes the neighbors of node $\bm{v}$,
$\bm{H}^{(k)}$ is the representation vector of the node $\bm{v}$ at layer $k$,
and when $k=0$, $\bm{H}^{(0)}$ is initialized with the input node features,
which is extracted from $\bm{X}$. $combine$ and $aggregate$ are functions with
learnable parameters. After $K$ rounds of massage passing, a readout function
will pool the node representations to obtain the graph representation
$\bm{h}_{i}$ for $\bm{G}_{i}$:

$\displaystyle{\bm{h}_{i}}=readout\Big{(}\bm{H}_{\bm{v}},\bm{v}\in\bm{V}_{i}\Big{)}.$
(2)

#### Contrastive learning.

We follow the preliminaries of contrastive learning Tian et al. (2020):
learning an embedding that maximizes agreement between the original and
augmented features of the same sample, namely positives, and separates the
features of different samples, namely negatives, in latent space. We denote
$\bm{G}^{\prime}_{i}$ is the augmented graph of $\bm{G}_{i}$.

To impose contrastive learning, we feed the inputs $\bm{G}_{i}$ and
$\bm{G}^{\prime}_{i}$ into the encoder $f(\cdot)$ to learn representations
$\bm{h}_{i}$ and $\bm{h}_{i}^{\prime}$, and the representations are further
mapped into features $\bm{z}_{i}$ and $\bm{z}_{i}^{\prime}$ by a projection
head $g(\cdot)$ Chen et al. (2020). The encoder $f(\cdot)$ and projection head
$g(\cdot)$ are trained by contrasting the features, and the loss Oord et al.
(2018) is formulated as follows:

${\mathcal{L}_{contrast}}=-\log\frac{\exp\left(\frac{<\bm{z}^{+}>}{\tau}\right)}{\exp\left(\frac{<\bm{z}^{+}>}{\tau}\right)+\sum{\exp\left(\frac{<\bm{z}^{-}>}{\tau}\right)}}$
(3)

where $\bm{z}^{+}$ denotes the pair of $\\{\bm{z}_{i}^{\prime},\bm{z}_{i}\\}$,
$\bm{z}^{-}$ is a set of pairs, i.e.,
$\Big{\\{}\\{\bm{z}_{j}^{\prime},\bm{z}_{i}\\},\\{\bm{z}_{i}^{\prime},\bm{z}_{j}\\}\Big{|}j\in\llbracket{1,N}\rrbracket,j\neq
i\Big{\\}}$, $<\cdot>$ denotes a discriminating function to measure the
similarity of features, and $\tau$ is the temperature factor Chen et al.
(2020). Note that, after training is completed, the projection head $g(\cdot)$
is discarded, and the representations are directly used for downstream tasks.

###  3.2 Meta Graph Augmentation

Different from benchmark methods that randomly dropping or adding edges to
augment a graph Hassani and Khasahmadi (2020) Wan et al. (2020), we motivate
our method to impose graph augmentation in an learnable manner Suresh et al.
(2021). We rethink the learning paradigm of contrastive learning and find that
such an approach relies heavily on hard and informative features in training.
Therefore, to boost the performance on downstream tasks of the learned
representations, we propose to use a trick of meta-learning to generate
informative graph augmentation, which is to guide the encoder to mine
discriminative knowledge from graphs.

#### Learnable graph augmentation.

As the architecture shown in Figure 2, we propose a learnable approach to
augment graph. In detail, suppose $\bm{A}$ is the adjacency matrix of
$\bm{G}_{i}$ where the initial weights of the connected nodes are set to $1$
and others are valued by $0$. $\widetilde{\bm{A}}=\bm{A}+\bm{I}$, where
$\bm{I}$ denotes the self-connection of each node $\bm{e}\in\bm{E}$. We use a
neural network $a(\cdot)$, as the LGA (see Appendix for the implementation),
to generate the augmented graph $\bm{G}_{i}^{\prime}$ from the original graph
$\bm{G}_{i}$, where $\widetilde{\bm{A}}_{i}^{\prime}$ is the adjacency matrix
with self-connections $\bm{I}$ of a graph $\bm{G}_{i}^{\prime}$. $\bm{G}_{i}$
and $\bm{G}_{i}^{\prime}$ are then encoded into features $\bm{z}_{i}$ and
$\bm{z}_{i}^{\prime}$ by the encoder $f(\cdot)$ and projection head
$g{(\cdot)}$.

Method | PROTEINS | MUTAG | DD | COLLAB | RDT-M5K | IMDB-B | IMDB-M
---|---|---|---|---|---|---|---
GIN RIU | 69.03$\pm 0.33$ | 87.61$\pm 0.39$ | 74.22$\pm 0.30$ | 63.08$\pm 0.10$ | 27.52$\pm 0.61$ | 51.86$\pm 0.33$ | 32.81$\pm 0.57$
InfoGraph | 72.57$\pm 0.65$ | 87.71$\pm 1.77$ | 75.23$\pm 0.39$ | 70.35$\pm 0.64$ | 51.11$\pm 0.55$ | 71.11$\pm 0.88$ | 48.66$\pm 0.67$
GraphCL | 72.86$\pm 1.01$ | 88.29$\pm 1.31$ | 74.70$\pm 0.70$ | 71.26$\pm 0.55$ | 53.05$\pm 0.40$ | 70.80$\pm 0.77$ | 48.49$\pm 0.63$
AD-GCL | 73.46$\pm 0.67$ | 89.22$\pm 1.38$ | 74.48$\pm 0.62$ | 72.90$\pm 0.83$ | 53.15$\pm 0.78$ | 71.12$\pm 0.98$ | 48.56$\pm 0.59$
MEGA-IL | 73.89$\pm 0.62$ | 90.34$\pm 1.20$ | 75.78$\bf\pm 0.63$ | 73.54$\pm 0.82$ | 53.16$\pm 0.65$ | 71.08$\pm 0.73$ | 49.09$\pm 0.79$
MEGA | 74.20$\bf\pm 0.73$ | 91.10$\bf\pm 1.34$ | 75.56$\pm 0.63$ | 73.96$\bf\pm 0.73$ | 54.32$\bf\pm 0.79$ | 71.95$\bf\pm 0.98$ | 49.52$\bf\pm 0.62$
Table 1: Performance of classification accuracy on datasets from TU Dataset (Averaged accuracy $\pm$ std. over 10 runs). We highlight the best records in bold. Method | molesol | mollipo | molbbbp | moltox21 | molsider
---|---|---|---|---|---
| Regression tasks (RMSE $\downarrow$) | Classification tasks (ROC-AUC % $\uparrow$)
GIN RIU | 1.706$\pm 0.180$ | 1.075$\pm 0.022$ | 64.48$\pm 2.46$ | 71.53$\pm 0.74$ | 62.29$\pm 1.12$
InfoGraph | 1.344$\pm 0.178$ | 1.005$\pm 0.023$ | 66.33$\pm 2.79$ | 69.74$\pm 0.57$ | 60.54$\pm 0.90$
GraphCL | 1.272$\pm 0.089$ | 0.910$\pm 0.016$ | 68.22$\pm 1.89$ | 72.40$\pm 1.01$ | 61.76$\pm 1.11$
AD-GCL | 1.270$\pm 0.092$ | 0.926$\pm 0.037$ | 68.26$\pm 1.32$ | 71.08$\pm 0.93$ | 61.83$\pm 1.14$
MEGA-IL | 1.153$\pm 0.103$ | 0.852$\pm 0.022$ | 68.34$\pm 1.38$ | 72.08$\pm 0.82$ | 63.37$\bf\pm 0.87$
MEGA | 1.121$\bf\pm 0.092$ | 0.831$\bf\pm 0.018$ | 69.71$\bf\pm 1.56$ | 72.45$\bf\pm 0.67$ | 62.92$\pm 0.76$
Table 2: Performance of chemical molecules property prediction in OGB
datasets. There are two kinds of tasks, regression tasks and classification
tasks. We highlight the best records in bold.

#### Auxiliary meta-learning.

In contrastive learning, we need hard and informative features to learn
discriminative representations. To this end, we build an LGA and train it by a
meta-learning approach. In training, we first fix LGA $a_{\sigma}(\cdot)$ and
train the encoder $f_{\phi}(\cdot)$ and the projection head
$g_{\varphi}(\cdot)$ by back-propagating $\mathcal{L}_{contrast}$, where
$\sigma$, $\phi$, and $\varphi$ denote the network parameters of $a(\cdot)$,
$f(\cdot)$, and $g(\cdot)$, respectively. Then, $f_{\phi}(\cdot)$ and
$g_{\varphi}(\cdot)$ are fixed, and $a_{\sigma}(\cdot)$ is trained by
computing its gradients with respect to the performance of $f_{\phi}(\cdot)$
and $g_{\varphi}(\cdot)$, and the meta-learning updating objective is as
follows:

$\mathop{\arg\min}_{\sigma}\bigg{(}\mathcal{L}_{MEGA}\Big{(}g_{\mathring{\varphi}}\big{(}f_{\mathring{\phi}}(\bm{G})\big{)},g_{\mathring{\varphi}}\big{(}f_{\mathring{\phi}}(a_{\sigma}(\bm{G}))\big{)}\Big{)}\bigg{)}$
(4)

where $g_{\mathring{\varphi}}\big{(}f_{\mathring{\phi}}(\bm{G})\big{)}$
denotes a set of the features extracted from original graphs,
$g_{\mathring{\varphi}}\big{(}f_{\mathring{\phi}}(a_{\sigma}(\bm{G}))\big{)}$
denotes a set that includes the features of augmented graphs, and
$a_{\sigma}(\bm{G})$ denotes $\bm{G}^{\prime}$. $\mathring{\phi}$ and
$\mathring{\varphi}$ represent the corresponding parameters of the encoder and
projection head, which are updated with one gradient back-propagation using
the contrastive loss defined in Equation 3:

$\displaystyle\mathring{\phi}=\phi-\ell\nabla_{\phi}\bigg{(}\mathcal{L}_{contrast}\Big{(}g_{\varphi}\big{(}f_{\phi}(\bm{G})\big{)},g_{\varphi}\big{(}f_{\phi}(\bm{G}^{\prime})\big{)}\Big{)}\bigg{)}$
(5)
$\displaystyle\mathring{\varphi}=\varphi-\ell\nabla_{\varphi}\bigg{(}\mathcal{L}_{contrast}\Big{(}g_{\varphi}\big{(}f_{\phi}(\bm{G})\big{)},g_{\varphi}\big{(}f_{\phi}(\bm{G}^{\prime})\big{)}\Big{)}\bigg{)}$

where $\ell$ is the learning rate shared between $\phi$ and $\varphi$. The
idea behind the meta updating objective is that we perform the second-
derivative trick Liu et al. (2019) to train $a_{\sigma}(\cdot)$. Specifically,
a derivative over the derivative (i.e., a Hessian matrix) of
$\\{\phi,\varphi\\}$ is used to update $\sigma$. We compute the derivative
with respect to $\sigma$ by using a retained computational graph of
$\\{\phi,\varphi\\}$. Then, $\sigma$ is updated by

$\displaystyle\sigma=\sigma-\ell^{\prime}\nabla_{\sigma}\bigg{(}\mathcal{L}_{MEGA}\Big{(}g_{\mathring{\varphi}}\big{(}f_{\mathring{\phi}}(\bm{G})\big{)},g_{\mathring{\varphi}}\big{(}f_{\mathring{\phi}}(\widehat{\bm{G}^{\prime}})\big{)}\Big{)}\bigg{)}$
(6)

where $\ell^{\prime}$ represents the learning rate of $\sigma$, and
$\widehat{\bm{G}^{\prime}}$ is the augmented graphs with stop-gradient, which
is defined as $\widehat{\bm{G}^{\prime}}=a_{\sigma}(\bm{G}).detach()$.
$\mathcal{L}_{MEGA}$ is to train $a_{\sigma}(\cdot)$ to generate hard and
informative augmented graphs defined as follows:

$\displaystyle\mathcal{L}_{MEGA}=\underbrace{tr(\bm{C})-de(\bm{C})}_{instance\
term}+\lambda\underbrace{\left(tr(|\mathbb{1}-\bm{D}|^{2})+de(|\bm{D}|^{2})\right)}_{feature\
term}$ (7)

where $tr(\cdot)$ denotes the matrix trace function, which is defined as
$tr(M)=\sum_{i}M_{ii}$, and $de(\cdot)$ is a matrix calculation function
defined as $de(M)=\sum_{i}\sum_{j\neq i}M_{ij}$. $|\cdot|^{2}$ presents a
matrix element-wise square function defined as $|M|^{2}=M\times M$ by Hadamard
product, and $\mathbb{1}$ presents an identity matrix. $\lambda$ is the
coefficient that controls the balance between two terms of
$\mathcal{L}_{MEGA}$. Intuitively, the $instance\ term$ aims to lead MEGA to
generate instance-level challenging examples for self-supervised learning.
Inspired by Zbontar et al. (2021), we design the $feature\ term$ to promote
the model to learn dimensionally non-redundant representations, respectively.
Concretely, minimizing the proposed $\mathcal{L}_{MEGA}$ by using the second-
derivative technique can guide $a_{\sigma}(\cdot)$ to generate hard and
informative augmented graphs. $\bm{C}$ denotes the cross-correlation matrix
computed between the features of the original graphs and augmented graphs in a
batch, as follows:

$\displaystyle\bm{C}_{ij}=\frac{\bm{z}_{i}\cdot\bm{z}_{j}^{\prime}}{|\bm{z}_{i}|\cdot|\bm{z}^{\prime}_{j}|}$
(8)

where $i,j\in\llbracket{1,N}\rrbracket$ in a batch $N$ of graphs. $\bm{D}$ is
the cross-correlation matrix computed between the multi-dimensional features
of the original graphs and the corresponding augmented graphs along the batch,
which is defined as:

$\displaystyle\bm{D}_{pq}=\frac{\sum_{i}(\bm{z}_{i,p}\cdot\bm{z}_{i,q}^{\prime})}{\sqrt{\sum_{i}(\bm{z}_{i,p})^{2}}\cdot\sqrt{\sum_{i}(\bm{z}^{\prime}_{i,q})^{2}}}$
(9)

where $i\in\llbracket{1,N}\rrbracket$ indexes batch graphs and
$p,q\in\llbracket{1,N^{D}}\rrbracket$ index the feature dimension of the
original graph and the corresponding augmented graph, and $N^{D}$ denotes the
number of feature dimension. $\bm{C}$, $\bm{D}$ aim to train
$a_{\sigma}(\cdot)$ to generate hard and informative augmented graphs,
respectively. Concretely, the objective of auxiliary meta-learning is to
enable LGA to learn augmented graphs that are hard and informative for the
encoder, thereby improving the encoder’s learning process for the next
iteration.

##  4 Experiments

(a) MUTAG

(b) PROTEINS

(c) IMDB-B

Figure 3: Results of MEGA’s performance with a range of factor $\lambda$. We
perform MEGA on three benchmark datasets: MUTAG, PROTEINS, and IMDB-B. The
abscissa axis represents the value of $\lambda$, and the ordinate axis
represents the accuracies. Figure 4: This figure shows the visualized output
graph features on the MUTAG dataset. The graph features are projected into a
color image in RGB format, where different colors represent different types of
features. The abscissa axis represents the output feature dimensions of
compared methods, and the ordinate axis represents graphs of different
classes.

In this section, we demonstrate the effectiveness of MEGA on various benchmark
datasets. Our experiments were conducted in an unsupervised learning setting.

###  4.1 Comparison with State-of-the-art Methods

Datasets. We evaluate our method on twelve benchmark datasets in two major
categories: 1) Social Networks: RDT-M5K, IMDB-B, IMDB-M from TU Dataset Morris
et al. . 2) Molecules: PROTEINS, MUTAG, COLLAB and DD from TU Dataset Morris
et al. and molesol, mollipo, molbbbp, moltox21 and molsider from Open Graph
Benchmark (OGB) Hu et al. (2020b).

#### Experiment settings.

We compared MEGA with four unsupervised/self-supervised learning baselines,
which include randomly initialized untrained GIN (GIN RIU) Xu et al. (2018),
InfoGraph Sun et al. (2020), GraphCL You et al. (2020) and AD-GCL Suresh et
al. (2021). Experiment results of InfoGraph Sun et al. (2020) and GraphCL You
et al. (2020) show that they generally outperform graph kernel and network
embedding methods including Kriege et al. (2020), Grover and Leskovec (2016),
and Adhikari et al. (2018). As discussed in the method section, the instance-
level constrains and feature-level constraints of $\mathcal{L}_{MEGA}$ are
balanced by parameter $\lambda$. To study the effects of these constraints, we
set $\lambda=0$ for the ablation study, termed MEGA-IL. We followed the
experimental protocol of AD-GCL, including the train/validation/test splits.
The average classification accuracy with standard deviation on the test
results over the last ten runs of training is reported. For a fair comparison,
we adopted GIN as the encoder as other baselines do. We adopt the Adam
optimizer with a learning rate of $10^{-4}$ for learnable graph augmentation
and a learning rate of $10^{-3}$ for graph encoding. We use 50 training epochs
on all datasets. All methods adopt a downstream linear classifier or regressor
with the same hyper-parameters.

#### Results.

The results are reported in Table 1 and 2. The results show that MEGA achieves
the best results compared with baselines across benchmark datasets. We
attribute such performance to MEGA’s abilities to generate both hard and
informative augmented graph features. The results show that MEGA outperforms
MEGA-IL across most datasets, which proves that the feature-level constraints
do improve the network to learn informative representations. MEGA-IL still
performs better than most of the baselines that adopt the same encoder and
contrastive learning pattern, which means that the instance-level constrains
of $\mathcal{L}_{MEGA}$ work well.

###  4.2 Evaluation of Feature-level Constrains

For further evaluation of feature-level constraints, we change the value of
$\lambda$ and observe how the performance changes. We adopt three different
datasets, including two molecule datasets and one social network dataset.

The results are reported in Figure 3. The performance changes as the factor
$\lambda$ changes. When $\lambda$ takes 0.1, the performance is optimal among
all tasks. The results prove that feature-level constraints can enhance the
discrimination of features to a certain extent. The feature-level constraints
ensure that the generated augmented graph correlates with the original graph,
preventing LGA from learning outrageous graph augmentation. However, if we
overly increase the impact of feature-level constraints, the generation of
hard augmented graph features could be interfered.

###  4.3 Analyze on Representations

To better understand the quality of the representations learned by MEGA, we
visualize the output graph features. For comparison, we conducted experiments
on four different networks: (1) UNTRAINED, a randomly initialized GIN encoder
without training. (2) CCL, a conventional graph contrastive learning network
without our proposed LGA. (3) MEGA-IL, MEGA without feature-level constraints.
(4) MEGA.

The results are shown in Figure 4. We intuitively observe the representations
of each graph and find that MEGA and MEGA-IL output more ”colorful” results
than RI-GIN and CCL, which indicates that their output is more informative. In
detail, for MEGA and MEGA-IL, there are many vertical lines of different
colors, which means that the difference between the dimensions of the feature
is significant. This phenomenon is more evident on MEGA, indicating that the
feature-level constraints make the feature dimensions less redundant.

##  5 Conclusions

This paper proposed a novel meta graph augmentation to boost the
representation ability of graph contrastive learning. We apply secondary
derivative technique to update a learnable graph augmenter, which is to
generate hard and informative augmented graph for contrastive learning. This
way, we can yield a representation with uniformity at the instance-level and
informativeness at the feature-level.

## Acknowledgements

The authors thank all the anonymous reviewers. This work is supported by the
Strategic Priority Research Program of the Chinese Academy of Sciences, Grant
No. XDA19020500.

## References

  * Adhikari et al. [2018] Bijaya Adhikari, Yao Zhang, Naren Ramakrishnan, and B Aditya Prakash.  Sub2vec: Feature learning for subgraphs.  In Pacific-Asia Conference on Knowledge Discovery and Data Mining, 2018.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning. PMLR, 2020.
  * Chuang et al. [2020] Ching-Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka.  Debiased contrastive learning.  arXiv preprint arXiv:2007.00224, 2020.
  * Ermolov et al. [2021] Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe.  Whitening for self-supervised representation learning.  In International Conference on Machine Learning, pages 3015–3024. PMLR, 2021.
  * Finn et al. [2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In Proceedings of the 34th ICML. PMLR, 2017.
  * Grill et al. [2020] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in Neural Information Processing Systems, 33:21271–21284, 2020.
  * Grover and Leskovec [2016] Aditya Grover and Jure Leskovec.  node2vec: Scalable feature learning for networks.  In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855–864, 2016.
  * Hassani and Khasahmadi [2020] Kaveh Hassani and Amir Hosein Khasahmadi.  Contrastive multi-view representation learning on graphs.  In International Conference on Machine Learning, pages 4116–4126. PMLR, 2020.
  * Hu et al. [2020a] W Hu, B Liu, J Gomes, M Zitnik, P Liang, V Pande, and J Leskovec.  Strategies for pre-training graph neural networks.  In International Conference on Learning Representations (ICLR), 2020\.
  * Hu et al. [2020b] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec.  Open graph benchmark: Datasets for machine learning on graphs.  Neural Information Processing Systems (NeurIPS), 2020.
  * Kipf and Welling [2016] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  arXiv preprint arXiv:1609.02907, 2016.
  * Kriege et al. [2020] Nils M Kriege, Fredrik D Johansson, and Christopher Morris.  A survey on graph kernels.  Applied Network Science, 5(1):1–42, 2020.
  * Liu et al. [2019] Shikun Liu, Andrew Davison, and Edward Johns.  Self-supervised generalisation with meta auxiliary learning.  Advances in Neural Information Processing Systems, 32, 2019.
  * [14] Christopher Morris, Nils M Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann.  Tudataset: A collection of benchmark datasets for learning with graphs.  In ICML 2020 Workshop on Graph Representation Learning and Beyond.
  * Oord et al. [2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * Schmidhuber [2014] Jürgen Schmidhuber.  Learning complex, extended sequences using the principle of history compression.  Neural Computation, 4(2):234–242, 2014.
  * Sun et al. [2020] Fan-Yun Sun, Jordon Hoffman, Vikas Verma, and Jian Tang.  Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization.  In International Conference on Learning Representations, 2020.
  * Suresh et al. [2021] Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville.  Adversarial graph augmentation to improve graph contrastive learning.  Advances in Neural Information Processing Systems, 34, 2021.
  * Tian et al. [2020] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16, pages 776–794. Springer, 2020.
  * Veličković et al. [2017] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.  Graph attention networks.  arXiv preprint arXiv:1710.10903, 2017.
  * Veličković et al. [2018] Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm.  Deep graph infomax.  arXiv preprint arXiv:1809.10341, 2018.
  * Wan et al. [2020] Sheng Wan, Shirui Pan, Jian Yang, and Chen Gong.  Contrastive and generative graph convolutional networks for graph-based semi-supervised learning.  arXiv preprint arXiv:2009.07111, 2020.
  * Xu et al. [2018] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka.  How powerful are graph neural networks?  In International Conference on Learning Representations, 2018.
  * You et al. [2020] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen.  Graph contrastive learning with augmentations.  Advances in Neural Information Processing Systems, 33:5812–5823, 2020.
  * Zbontar et al. [2021] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In International Conference on Machine Learning, pages 12310–12320. PMLR, 2021.