longtable

## Improving accuracy of GPT-3/4 results on biomedical data using a retrieval-
augmented language model

David Soong*1, Sriram Sridhar*1, Han Si1, Jan-Samuel Wagner1, Ana Caroline
Costa Sá1, Christina Y Yu1, Kubra Karagoz1, Meijian Guan1, Hisham Hamadeh1,
Brandon W Higgs1

1Data Sciences, Genmab, Princeton, NJ

*Equal contribution

## Abstract

Large language models (LLMs) have made significant advancements in natural
language processing (NLP). Broad corpora capture diverse patterns but can
introduce irrelevance, while focused corpora enhance reliability by reducing
misleading information. Training LLMs on focused corpora poses computational
challenges. An alternative approach is to use a retrieval-augmentation (RetA)
method tested in a specific domain.

To evaluate LLM performance, OpenAI's GPT-3.5, GPT-4, Bing's Prometheus, and a
custom RetA model were compared using 19 questions on diffuse large B-cell
lymphoma (DLBCL) disease. Eight independent reviewers assessed responses based
on accuracy, relevance, and readability (rated 1-3).

The RetA model performed best in accuracy (12/19 3-point scores, total=47) and
relevance (13/19, 50), followed by GPT-4 (8/19, 43; 11/19, 49). GPT-4 received
the highest readability scores (17/19, 55), followed by GPT-3.5 (15/19, 53)
and the RetA model (11/19, 47). Prometheus underperformed in accuracy (34),
relevance (32), and readability (38).

Both GPT-3.5 and GPT-4 had more hallucinations in all 19 responses compared to
the RetA model and Prometheus. Hallucinations were mostly associated with non-
existent references or fabricated efficacy data.

These findings suggest that RetA models, supplemented with domain-specific
corpora, may outperform general-purpose LLMs in accuracy and relevance within
specific domains. However, this evaluation was limited to specific questions
and metrics and may not capture challenges in semantic search and other NLP
tasks. Further research will explore different LLM architectures, RetA
methodologies, and evaluation methods to assess strengths and limitations more
comprehensively.

##

## Introduction

The development of large language models (LLMs), such as bidirectional encoder
representations from transformer (BERT) and generative pre-trained transformer
(GPT), has revolutionized the field of natural language processing [1], [2],
[3] [4]. Applications of these LLMs have ranged from sentiment analysis and
machine translation to code generation and question answering in several
domains [5-10] – all demonstrating remarkable performance. However, despite
their impressive execution and widespread use, LLMs do not know the
information they were not trained on, and often lack domain-specific knowledge
and vocabulary. They can also perpetuate biases based on skewed content in the
training data, and need to be further refined through reinforcement learning
and alignment approaches to understand user intentions while making them more
truthful and less toxic [11, 12]. Furthermore, concerns have been raised about
the potential for LLMs to generate hallucinated or misleading information,
which can have severe implications in scientific research and led to the
critical determinants of distinguishing fact from fiction leading to
discontinuation of, as was the case for Meta’s Galactica [13, 14].

Popular LLMs with billions of parameters such as GPT-3 [4] , PaLM [15], OPT
[16], and LLaMA [17] are typically trained on vast amounts of information
collected from the Internet (e.g. the Common Crawl dataset [18]) and capture a
diverse range of language patterns and knowledge. Word and sentence embeddings
are high-dimensional numerical representations of concepts scaled by the size
of the corpus and complexity of language usage [19] [20, 21]. This can produce
a higher level of generality and flexibility in the model's ability to yield
natural language, making it more robust and adaptable to a range of
applications. Similarly, a broad corpus can capture the diversity of language
usage across different domains and genres. For example, a model trained on a
broad corpus could potentially generate natural language in scientific
literature, social media, or news articles, with equal ease [3].

Nonetheless, a wide-ranging corpus can inadvertently incorporate a significant
amount of noise or irrelevant data, resulting in a reduced signal-to-noise
ratio [22]. This may adversely affect the generated text's quality, leading to
decreased coherence, meaning, or accuracy. Additionally, biases and
inaccuracies may arise in the model's comprehension of natural language. A
corpus that predominantly features one type of language or cultural context
may display bias towards that specific domain or culture [23]. Although a
corpus may strive to encompass a diverse range of domains, the sheer vastness
of the domain space makes it currently unfeasible to include all relevant
domains. Moreover, as more domains are incorporated, there is a risk that LLMs
trained on such a comprehensive corpus may struggle to differentiate language
from various domains, particularly when faced with prompts that lack
sufficient context.

One approach to address these limitations is to retrain or finetune an LLM
with a focused corpus tailored to a specific domain or application [22] [24]
thereby reducing the risk of generating irrelevant or misleading information
and enhancing the reliability and precision of the LLM's outputs in
specialized contexts. Numerous publications have highlighted the efficacy of
domain specific LLMs in their respective fields. For example, BioBERT [25]
targets biomedical text mining tasks, SciBERT [26] and PubMedBERT [27] address
scientific literature, and Legal-BERT [28] specializes in legal text
processing. These approaches minimize noise and irrelevant information in the
text, potentially reducing hallucinations.

However, retraining LLMs to encompass new documents might be impractical due
to the cumulative computational costs and data scientist resources required
per update. The LLM architecture might also need to be updated to incorporate
more parameters to memorize more facts [29]. As LLMs have demonstrated
extraordinary abilities to learn in-context information purely from its prompt
[4], RetA approaches have proven promising [24] [30]. These models first
retrieve relevant context from domain-specific corpora based on a user query
using lexical search (e.g. BM25 [31]) or a pretrained/fine-tuned semantic
retriever (e.g. Spider[32], OpenAI embeddings [33]), and then seed a pre-
trained LLM with such context to provide grounded answers while avoiding the
prohibitive time and cost of retraining an LLM.

In this study, several LLMs were evaluated to investigate if a retrieval-
augmentation approach on a focused corpus could improve the accuracy of LLMs
applications in biomedical Q&A. Three scoring metrics were utilized to
objectively compare outputs between models using a set of evaluation-based
questions focused on disease characterization, genetic subtypes, treatment
options, and clinical outcomes in diffuse large B-cell lymphoma DLBCL. These
observations provide insights into the pros and cons of each LLM and suggest
potential areas for improvement to meet utility requirements for rigorous drug
development and scientific research.

## Methods

_Evaluation framework_

The performance of generically trained LLMs was tested versus a RetA LLM in
question answering (Q&A) tasks related to disease biology and drug
development. A set of 19 questions focused on mechanisms and treatments
associated with DLBCL were provided to evaluate LLM performance. The questions
covered a broad range of topics related to DLBCL disease biology including
clinical and molecular subtypes, genetic subsets and relevant biomarkers,
clinical management, and standards of care and other available therapies.
Questions were designed to look for both qualitative and quantitative answers
(e.g. overall response rate and prevalence of genomic alterations). Each
question was provided to four different LLMs: Open AI’s general ChatGPT-3.5
[34], OpenAI’s general GPT-4 [34], Bing’s Prometheus model (referred to in
this manuscript as Bing chat, based on GPT-4 [35]), and a RetA LLM (based on
GPT-3) using a custom set of full-text publications associated with DLBCL
(Table 1). The questions intentionally varied in detail to assess the ability
of each LLM to infer the expected result. For example, question #15 provided a
concise query for DLBCL diagnosis and prognosis, while question #3 asked
specific treatments for a target in the disease with accompanying references
to support the answer.

The two general GPT-based LLMs from OpenAI were only trained on content up to
September 2021 (OpenAI GPT-4 Technical Report [36]), as opposed to Bing’s
Prometheus and the RetA models. Release versions of GPT-4 and GPT-3.5 used to
answer the questions were from 3/23/23 to 4/28/23 (updates were released on a
weekly or bi-weekly basis and were documented).

_RetA model and dataset_

Scientific papers were downloaded from PubMed Central (PMC [37]) using the
Entrez E-utilities [38]). Each of the following queries was used to retrieve
up to 500 articles: ‘diffuse large b-cell lymphoma’, ‘follicular lymphoma’,
‘epcoritamab’, ‘glofitamab’, ‘minimal residual disease’, ‘ctDNA’. By default,
Entrez returns articles sorted by PMC identifier. The queries used were meant
to generate a corpus specific to DLBCL, related biomarkers, standards of care,
and therapeutic options, not to specifically answer the questions used in this
evaluation. This created a unique dataset of 1,868 full-text articles. The
documents were first pre-processed to exclude potentially unstructured or
noisy text (e.g. figures, tables, references, author disclosure) and split
into segments of 4,000 tokens. Embeddings were then calculated using the
OpenAI model text-embedding-ada-002 and stored in a local database. When the
user entered a question, the query was transformed into an embedding vector
and compared to the database of embeddings using cosine similarity. The top
_k_ document segments by similarity were retrieved and formed the knowledge
context for the user query. The synthesis of the answer to the query was
achieved in two stages: in stage one, text-davinci-003 was used to answer the
query using each of the _k_ context segments with prompt instructions to
minimize inclusion of non-factual information from the LLM. This generated _k_
answers which were combined into a final response in the second stage using a
call to text-davinci-003 with a summarization prompt (Figure 1, Tables 2a,b).

_Evaluation metrics_

Answers were scored for each question on a three-point scale (1-3, with 3
being highest) based on three metrics: accuracy, relevance, and readability by
eight independent reviewers (Table 3), with each reviewer scoring a subset of
questions. Answers to all questions were searchable. Accuracy and relevance
assessments focused on factual correctness of answers, correctness of
references or links to references, or general pieces of knowledge included or
not included in an answer. The 3-point scale used for each evaluation category
also allowed for some granularity in scoring answers. For example, an answer
might be given a score of “2” if the result was factually correct but links to
supporting references were broken or incorrect. An answer which does not
directly address the question being asked or contains factually incorrect
information (i.e. hallucinations) might garner a score of “1” for accuracy. As
both the language model and oncology therapeutics fields are constantly
evolving, there is some recency bias associated with answers to questions and
the data which LLMs are trained on. This was in part accounted for through the
types of questions chosen and the scale used to assess responses. An emphasis
of the evaluation was to specifically look for factually incorrect answers, as
opposed to incomplete answers which may be a result of recency bias. Reviewers
were all Ph.D. level scientists with an average of 8 years of biopharma
industry experience and 11 years of post-doctoral work experience. All scores
were then assessed by one reviewer from the group to adjust for reviewer
biases. The prompts were stratified into three high level categories based on
relevance to drug information, disease biology, and clinical information.
Prompts were also grouped based on being general (i.e. high level) or specific
(i.e. asking for details) questions to better attribute subfield performance
within DLBCL in comparisons between LLMs.

## Results

Overall, the performance of the LLMs varied widely across the different
questions and metrics. In terms of accuracy, the RetA model of GPT-3 on DLBCL
publications outperformed the other LLMs with the highest (3-point) scoring
answers on 12/19 questions. GPT-4 was the next best performer with 3-point
scores on 8/19 questions. Bing’s Prometheus had 7/19 3-point scores for
accuracy while GPT-3.5 had the fewest high scoring answers (4/19 3-point
scores) (Figures 2, 3). The summated scores for accuracy showed that the RetA
model scored slightly higher than GPT-4 in the categories of drug and clinical
information (Figure 4). Bing’s Prometheus model did not perform well in
accuracy compared to all other models with low (1-point) scores on 10/19
questions (Figures 2, 3). This was primarily due to misrepresentation of
references in its answers. Conversely, GPT-4 and the RetA model had the fewest
low scoring answers for accuracy (1/19 and 3/19 respectively) across prompts
(Figures 2, 3).

Interestingly, Bing’s Prometheus model was the only one to not score a value
of 1 in accuracy for question #6 (“What is the overall response rate of DLBCL
patients treated with glofitamab?”). Numerical overall and complete response
rates (ORR and CRR, respectively) reported by GPT-3.5 (ORR=65.1%, CRR=35.1%)
and GPT-4 (ORR=62.7%, CRR=39.2%) were not consistent with their references
cited and had either fabricated or provided incorrect references. Bing’s
Prometheus model scored a value of 2 because there was a mixture of accurate
and inaccurate answers to the question, i.e., this model accurately captured
the ORR value of glofitamab treatment (52%) in Dickinson et al, NEJM reference
[39], but also incorrectly used the median duration of objective response
rather than median duration of CR. The RetA model result was not accurate in
answering this question because the official glofitamab trial efficacy paper
[39] was not available on PubMed Central (https://www.ncbi.nlm.nih.gov/pmc/)
and therefore not included in the corpus.

In terms of relevance, the RetA model performed slightly better than GPT-4 and
GPT-3.5. The RetA model scored high (3-point) on 13/19 questions, compared to
11/19 and 10/19 in GPT-4 and GPT-3.5 respectively. Bing’s Prometheus model
performed worst in this category with scores of 1 in 8/19 questions (Figures
2, 3). The other three LLMs had few-to-no low scoring answers to prompts with
respect to relevance. The irrelevant answers (i.e. low scoring questions)
across all LLMs were primarily due to references to other diseases or
treatment. For example, in question #14 (“Have checkpoint inhibitor treatments
in monotherapy or combination therapy settings shown efficacy in DLBCL
patients? Provide references.”), the GPT-4 model cited three references, one
of which was in Hodgkin’s lymphoma (DLBCL is a non-Hodgkin’s lymphoma) and
another that discussed CAR-T, which is not a checkpoint inhibiting drug agent,
though the model associated this treatment modality with immunotherapies and
extended relevance to CAR-T therapies. GPT-3.5 also cited a reference
evaluating a checkpoint inhibitor treatment in Hodgkin’s lymphoma.

Finally, for readability, GPT-4 scored the highest with 17/19 scores of 3,
followed by GPT-3.5 with 15/19, and the RetA model with 11/19 (Figures 2, 3).
The summated scores demonstrated parity between GPT-4 and GPT-3.5 across all
categories (Figure 4). Readability was particularly low scoring in the
clinical category of questions for the RetA. Bing’s Prometheus model once
again scored last in this category (7/19 3-point scores), primarily due to
concise, yet vague answers, often with little detail. For example, for
question #7 (“What is a treatment to use in DLBCL patients who have progressed
on CAR-T?”), Bing’s Prometheus model simply reported references without
summarization, including one study where multiple drugs were approved, and
referenced only those of approved agents, ignoring studies evaluating
investigational drug agents.

Across the 19 questions, both GPT-3.5 and GPT-4 LLMs generated a considerably
higher number of hallucinations in their responses (31 from 13 questions and
19 from 8 questions, respectively) compared to the RetA model and Bing’
Prometheus model (3 from 3 questions and 2 from 1 question, respectively).
These were primarily associated with fabrication of both references and
clinical results. Although LLMs are known to be behind in mathematical
capabilities [40], the inaccuracy of numerical results appeared to be due to
hallucinations or context understanding rather than limitations in
mathematical reasoning.

These results suggest that the performance of LLMs can vary widely depending
on the specific task and domain, though the RetA model enhanced with domain-
specific data may outperform more general-purpose LLMs in accuracy and
relevance. However, it should be noted that this evaluation was limited to a
specific set of questions and metrics, and further research is needed to fully
understand the strengths and limitations of different LLMs for semantic search
and other natural language processing tasks.

## Discussion

The advantages and drawbacks of using LLMs trained on broad corpora versus a
RetA approach ultimately depend on the specific use case and desired outcomes.
In biomedical and healthcare research, it is paramount to have accurate,
relevant, and unbiased information supported by published literature. In this
study, quantifying the accuracy and utility of LLMs was conducted for
answering qualitative and quantitative biomedical questions related to the
treatment and prognosis of patients with DLBCL. Results here demonstrated that
the RetA LLM performed better on biomedical-specific tasks than the other LLMs
evaluated, specifically with respect to accuracy of results. This suggests
that RetA LLMs can provide more accurate and reliable information for specific
fields, reducing the likelihood of generating irrelevant or misleading
outputs, while maintaining the flexibility and adaptability of a general LLM.

One major advantage of the RetA model is the easy integration of new domain
knowledge that the base LLM was not trained on. When a new document is added
to the corpus, the model only needs to calculate the embeddings to facilitate
retrieval during future queries. On the other hand, fine-tuning or retraining
an LLM on a new corpus takes both time and resources, and may not always be
possible depending on the choice of LLM – as of the publication of this study,
OpenAI has not offered an option to fine-tune their ChatGPT models; Meta’s
LLaMA model is also not available for commercial applications [17, 41].

However, since the RetA model needs to prompt a pre-trained LLM into
performing specific tasks such as summarizing across relevant documents and
extracting information without using prior knowledge, the model typically uses
a large amount of tokens as input and multiple iterations of base LLM
inference (i.e. text-completion API) calls, which can increase the compute
cost in its application. The dependence on a certain LLM (e.g. OpenAI GPT-3)
also implies that the desired prompt behavior needs to be closely monitored
when the LLM backend is updated with new training data, or when the user
switches to a different base LLM (e.g. GPT-4, Dolly 2 [42], Open Assistant
[43], or RedPajama [44]).

Furthermore, its performance is also bound by the limitations of the base
LLM’s vocabulary (tokenizer) and internal representation of concepts
(embedding). For example, question #13 asked about minimal residual disease
(MRD) in DLBCL, but the document retriever returned articles about MRD in
multiple myeloma and chronic lymphocytic leukemia \- two distinct
hematological malignancies from DLBCL. The RetA model relies on GPT-3 as the
summarization engine which failed to distinguish between the different disease
types, leading to an incorrect answer. These issues may be ameliorated by
utilizing more sophisticated document retrieval methods. For biomedical
literature, domain specific models such as BioBERT and PubMedBERT can be used
for tokenization and embedding calculation; additional metadata filters can
also be used to improve relevance of retrieved documents. As an example, when
the retrieval method was modified in the RetA model to directly search for
supporting articles on PubMed by significance, the model provided informative
and relevant answers detailing the measurement of disease clones with V(D)J
sequences, as well as the association with clinical outcomes.

Overall, general LLMs provide highly readable and coherent text in various
subjects. Furthermore, the performance of the RetA model demonstrated the
utility of using LLMs as a backend in performing various reasoning tasks
through specifically crafted prompts. Indeed, prompt engineering has been an
active area of research that continues to expand the capability of pre-trained
LLMs through methods such as: zero-shot [45], few-shot [4], chain of thought
[46], self-ask [47], and ReAct [48] reasoning. These reasoning properties
allow LLMs to be used as programmable agents to orchestrate and perform tasks
across different modalities or domains (e.g. ToolFormer [49], Visual ChatGPT
[50], Langchain [51], GPT plugins [52]).

Though findings here are informative, this study had several limitations that
need to be considered. First, the assessment included only 19 questions, which
accounted for various clinical, therapy, and biological content, which was an
attempt to address pertinent context in biomedical research, though certainly
not exhaustive. Second, the focus was on a single disease (DLBCL), which may
not be generalizable to other diseases or domains. Third, the scoring metrics
selected included accuracy, readability, and relevance, which might not have
captured other important aspects of the text such as strength, completeness,
and consistency. The scoring was performed across the entire answer as opposed
to by sentence or phrase within an answer. While scoring questions in this
manner can be subjective, we adjusted for this by using multiple reviewers and
having an additional overarching review to calibrate scores across questions.
There was also a range in experience among reviewers to account for any bias
associated with experience. Questions were also specific enough such that
available literature could be used to assess accuracy of answers. A point of
emphasis for the evaluation of responses was to look for factually incorrect
answers (hallucinations), which were more likely to garner the lowest score,
as opposed to answers which were factually correct but not exhaustive. Last,
the RetA model included an arbitrary number of full-text articles (1,868),
prioritized by PMC identifier, which might not have represented the most
relevant or comprehensive set of articles for the disease. It is possible that
an optima of accuracy, relevance, and readability can be achieved with an RetA
model by increasing the size and breadth of the corpus, and future work will
be needed to test this hypothesis. Despite these limitations, this study
provides valuable insights into the performance of LLMs on different types of
corpora and highlights the importance of domain-specific knowledge in
achieving higher accuracy and relevance.

With the rapid advancement and development of foundation models across text,
image, video and other data modalities, adaptation of AI in a fair, accurate,
and reliable fashion can make an immediate impact on healthcare and drug
development. In this study, focus was on evaluation of pre-trained and RetA
LLMs for biomedical Q&A in the field of clinical drug development. Future
research could explore methods incorporating biomedical ontology, knowledge
graphs, as well as other agent-based approaches to further enhance the
performance of LLMs [51], [52]. As open-source initiatives democratize AI
research [53, 54][42] [43] [44] and new emerging methodologies [55-58] begin
to offer possibilities to build custom LLMs with reduced compute resources and
time requirements, further integrating with multi-modal approaches that
leverage across molecular (e.g. mutations and gene expression), imaging
(pathology and radiology), electronic health records, and wearable sensor data
will provide a deeper understanding of disease biology and accelerate drug
development in a fair and socially responsible way [59] [60].

## Acknowledgements

The authors thank Bryan Ho and Swathi Vangala for their contributions to
infrastructure.

## Figures and Tables

Figure 1. Components and workflow of a RetA LLM. The pre-processing stage
splits documents into smaller chunks, creates embeddings and stores them in a
database. At the querying stage, a document retriever finds the most relevant
documents in the embeddings database, iteratively seeds the base LLM with
context to generate a response.

Table 1. Questions used for LLM evaluation classified into group and scope
categories.

Question # |  Question |  Group |  Scope
---|---|---|---
1 |  What is epcoritamab? Please provide sources for your answer. |  Drug information |  General
2 |  What are the subtypes of DLBCL? Please provide sources for your answer. |  Disease biology |  General
3 |  What are the antibody therapies targeting CD20 for treatment of DLBCL? Please provide sources for your answer. |  Drug information |  General
4 |  What is the standard of care for treatment of DLBCL? |  Clinical information |  Specific
5 |  What are the approved drugs for treatment of DLBCL? |  Clinical information |  Specific
6 |  What is the overall response rate of DLBCL patients treated with glofitamab? |  Clinical information |  Specific
7 |  What is a treatment to use in DLBCL patients who have progressed on CAR-T? |  Drug information |  General
8 |  What are common treatments used in patients who have relapsed or were refractory to standard of care treatments in DLBCL? |  Drug information |  General
9 |  Do any DLBCL patient subtypes respond more favorably to chemotherapy or CAR-T treatments? |  Clinical information |  Specific
10 |  What are the most common adverse events observed in DLBCL patients treated with R-CHOP? |  Clinical information |  Specific
11 |  What biomarkers in DLBCL have been reported to correlate with either response or progression following treatment with R-CHOP? |  Clinical information |  Specific
12 |  What treatment combinations have been shown to be effective in DLBCL patients who have progressed on CAR-T treatment? Please provide sources for your answer. |  Clinical information |  Specific
13 |  How can minimal residual disease (MRD) be used to understand clinical outcomes in DLBCL patients? Please provide sources for your answer. |  Disease biology |  General
14 |  Have checkpoint inhibitor treatments in monotherapy or combination therapy settings shown efficacy in DLBCL patients? Provide references. |  Drug information |  Specific
15 |  DLBCL diagnosis and prognosis. |  Clinical information |  General
16 |  Landscape of DLBCL treatment as SOC. Please provide sources for your answer. |  Clinical information |  Specific
17 |  Emerging novel treatment options for DLBCL patients. |  Drug information |  General
18 |  what is the importance of TP53 in DLBCL? |  Disease biology |  General
19 |  What is the prevalence of double hit mutations in lymphoma? |  Disease biology |  Specific

Table 2a. Prompts for GPT3 in the RetA workflow.

Stage |  Prompt
---|---
Stage one |  Instruction: You are a truthful AI assistant. You answer questions only based on provided context below. If the context is not relevant to the question, say you do not know the answer. No need to explain why. Context: {segment of article} Question: {user query} Answer:
Stage two |  Please combine the following paper's summaries. Only use the context below and not incorporate any prior knowledge. Paper #1: {answer 1 based on segment 1} Paper #2: {answer 2 based on segment 2}

Table 2b. Workflow and LLM descriptions used in this study.

Workflow |  Evaluation |  Base LLM
---|---|---
RetA LLM |  Python workflow |  text-davinci-003
chatGPT3.5 |  OpenAI web |  gpt-3.5-turbo
chatGPT4 |  OpenAI web |  gpt-4
BingChat |  Microsoft web |  Custom GPT4

Table 3. Answer scoring metric descriptions for LLM comparison.

|  Score
---|---
Metrics |  1 |  2 |  3
Accuracy |  Mostly inaccurate or misleading content |  A mix of accurate and inaccurate content |  Factually accurate and reliable content
Relevance |  Mostly irrelevant content |  Partially relevant content |  Highly relevant and on-point content
Readability |  Difficult to read, unclear or convoluted language |  Moderately readable, with some unclear passages |  Easy to read, clear, and concise language

Figure 2. Scores for each LLM within 3 metrics (accuracy, relevance,
readability) on a three-point scale. Questions are ordered by question
category (clinical, drug-related, disease-related). Question scope (general or
specific) is also annotated.

Figure 3. Count of scores (3-point, 2-point, and 1-point) across the 19
questions for each LLM in each score category (Accuracy, Readability,
Relevance).

Figure 4. Summarized scores for each LLM within 3 metrics (accuracy,
relevance, readability) and question categories (clinical information, disease
biology, and drug information).

## References

1\. Vaswani, A., et al. _Attention Is All You Need_. 2017. arXiv:1706.03762
DOI: 10.48550/arXiv.1706.03762.

2\. Devlin, J., et al. _BERT: Pre-training of Deep Bidirectional Transformers
for Language Understanding_. 2018. arXiv:1810.04805 DOI:
10.48550/arXiv.1810.04805.

3\. Radford, A., et al. _Language Models are Unsupervised Multitask Learners_.
2019.

4\. Brown, T.B., et al. _Language Models are Few-Shot Learners_. 2020\.
arXiv:2005.14165 DOI: 10.48550/arXiv.2005.14165.

5\. Chen, M., et al. _Evaluating Large Language Models Trained on Code_. 2021.
arXiv:2107.03374 DOI: 10.48550/arXiv.2107.03374.

6\. Gozalo-Brizuela, R. and E.C. Garrido-Merchan _ChatGPT is not all you need.
A State of the Art Review of large Generative AI models_. 2023\.
arXiv:2301.04655 DOI: 10.48550/arXiv.2301.04655.

7\. Phuong, M. and M. Hutter _Formal Algorithms for Transformers_. 2022\.
arXiv:2207.09238 DOI: 10.48550/arXiv.2207.09238.

8\. Services, B.P., _Introducing BloombergGPT, Bloomberg’s 50-billion
parameter large language model, purpose-built from scratch for finance_. 2023.

9\. Wu, S., et al. _BloombergGPT: A Large Language Model for Finance_. 2023.
arXiv:2303.17564 DOI: 10.48550/arXiv.2303.17564.

10\. Yang, X., et al., _A large language model for electronic health records._
NPJ Digit Med, 2022. 5(1): p. 194.

11\. Chung, H.W., et al. _Scaling Instruction-Finetuned Language Models_.
2022. arXiv:2210.11416 DOI: 10.48550/arXiv.2210.11416.

12\. Ouyang, L., et al., _Training language models to follow instructions with
human feedback._ ArXiv, 2022. abs/2203.02155.

13\. Heaven, W.D. _Why Meta’s latest large language model survived only three
days online_. 2022; Available from:
https://www.technologyreview.com/2022/11/18/1063487/meta-large-language-model-
ai-only-survived-three-days-gpt-3-science/.

14\. Taylor, R., et al. _Galactica: A Large Language Model for Science_. 2022.
arXiv:2211.09085 DOI: 10.48550/arXiv.2211.09085.

15\. Chowdhery, A., et al. _PaLM: Scaling Language Modeling with Pathways_.
2022. arXiv:2204.02311 DOI: 10.48550/arXiv.2204.02311.

16\. Zhang, S., et al. _OPT: Open Pre-trained Transformer Language Models_.
2022. arXiv:2205.01068 DOI: 10.48550/arXiv.2205.01068.

17\. Touvron, H., et al. _LLaMA: Open and Efficient Foundation Language
Models_. 2023. arXiv:2302.13971 DOI: 10.48550/arXiv.2302.13971.

18\. _Common Crawl_. Available from: https://commoncrawl.org/.

19\. Mikolov, T., et al. _Efficient Estimation of Word Representations in
Vector Space_. in _International Conference on Learning Representations_.
2013.

20\. Pennington, J., R. Socher, and C. Manning. _GloVe: Global Vectors for
Word Representation_. 2014. Doha, Qatar: Association for Computational
Linguistics.

21\. Reimers, N. and I. Gurevych. _Sentence-BERT: Sentence Embeddings using
Siamese BERT-Networks_. 2019. Hong Kong, China: Association for Computational
Linguistics.

22\. Gururangan, S., et al. _Don’t Stop Pretraining: Adapt Language Models to
Domains and Tasks_. 2020. Online: Association for Computational Linguistics.

23\. Bender, E.M., et al., _On the Dangers of Stochastic Parrots: Can Language
Models Be Too Big?_ , in _Proceedings of the 2021 ACM Conference on Fairness,
Accountability, and Transparency_. 2021, Association for Computing Machinery:
Virtual Event, Canada. p. 610–623.

24\. Guu, K., et al., _Retrieval Augmented Language Model Pre-Training_ , in
_Proceedings of the 37th International Conference on Machine Learning_ , D.
Hal, III and S. Aarti, Editors. 2020, PMLR: Proceedings of Machine Learning
Research. p. 3929–3938.

25\. Lee, J., et al., _BioBERT: a pre-trained biomedical language
representation model for biomedical text mining._ Bioinformatics, 2019. 36(4):
p. 1234-1240.

26\. Beltagy, I., K. Lo, and A. Cohan. _SciBERT: A Pretrained Language Model
for Scientific Text_. 2019. Hong Kong, China: Association for Computational
Linguistics.

27\. Gu, Y., et al., _Domain-Specific Language Model Pretraining for
Biomedical Natural Language Processing._ ACM Trans. Comput. Healthcare, 2021\.
3(1): p. Article 2.

28\. Chalkidis, I., et al. _LEGAL-BERT: The Muppets straight out of Law
School_. 2020. Online: Association for Computational Linguistics.

29\. Kaplan, J., et al. _Scaling Laws for Neural Language Models_. 2020\.
arXiv:2001.08361 DOI: 10.48550/arXiv.2001.08361.

30\. Ram, O., et al. _In-Context Retrieval-Augmented Language Models_. 2023.
arXiv:2302.00083 DOI: 10.48550/arXiv.2302.00083.

31\. Robertson, S. and H. Zaragoza, _The Probabilistic Relevance Framework:
BM25 and Beyond._ Found. Trends Inf. Retr., 2009. 3(4): p. 333–389.

32\. Ram, O., et al. _Learning to Retrieve Passages without Supervision_.
2022. Seattle, United States: Association for Computational Linguistics.

33\. Neelakantan, A., et al. _Text and Code Embeddings by Contrastive Pre-
Training_. 2022. arXiv:2201.10005 DOI: 10.48550/arXiv.2201.10005.

34\. OpenAI. _ChatGPT_. Available from: https://chat.openai.com/.

35\. Microsoft, _Bing._

36\. OpenAI _GPT-4 Technical Report_. 2023. arXiv:2303.08774 DOI:
10.48550/arXiv.2303.08774.

37\. NCBI. _PMC_. Available from: https://www.ncbi.nlm.nih.gov/pmc/.

38\. NCBI, _Entrez E-utilities._

39\. Dickinson, M.J., et al., _Glofitamab for Relapsed or Refractory Diffuse
Large B-Cell Lymphoma._ New England Journal of Medicine, 2022. 387(24): p.
2220-2231.

40\. Frieder, S., et al. _Mathematical Capabilities of ChatGPT_. 2023\.
arXiv:2301.13867 DOI: 10.48550/arXiv.2301.13867.

41\. Research, F. _llama github page_. Available from:
https://github.com/facebookresearch/llama.

42\. Mike Conover, M.H., Ankit Mathur, Xiangrui Meng, Jianwei Xie, Jun Wan,
Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia and Reynold Xin, _Free
Dolly: Introducing the World's First Truly Open Instruction-Tuned LLM_. 2023,
Databricks.

43\. _Open Assistant_. Available from: https://open-assistant.io/.

44\. _RedPajama, a project to create leading open-source models, starts by
reproducing LLaMA training dataset of over 1.2 trillion tokens_. 2023.

45\. Kojima, T., et al. _Large Language Models are Zero-Shot Reasoners_. 2022.
arXiv:2205.11916 DOI: 10.48550/arXiv.2205.11916.

46\. Wei, J., et al. _Chain-of-Thought Prompting Elicits Reasoning in Large
Language Models_. 2022. arXiv:2201.11903 DOI: 10.48550/arXiv.2201.11903.

47\. Press, O., et al. _Measuring and Narrowing the Compositionality Gap in
Language Models_. 2022. arXiv:2210.03350 DOI: 10.48550/arXiv.2210.03350.

48\. Yao, S., et al. _ReAct: Synergizing Reasoning and Acting in Language
Models_. 2022. arXiv:2210.03629 DOI: 10.48550/arXiv.2210.03629.

49\. Schick, T., et al. _Toolformer: Language Models Can Teach Themselves to
Use Tools_. 2023. arXiv:2302.04761 DOI: 10.48550/arXiv.2302.04761.

50\. Wu, C., et al. _Visual ChatGPT: Talking, Drawing and Editing with Visual
Foundation Models_. 2023. arXiv:2303.04671 DOI: 10.48550/arXiv.2303.04671.

51\. _LangChain_. Available from: https://python.langchain.com/en/latest/.

52\. _ChatGPT plugins_. Available from: https://openai.com/blog/chatgpt-
plugins.

53\. Biderman, S., et al. _Pythia: A Suite for Analyzing Large Language Models
Across Training and Scaling_. 2023. arXiv:2304.01373 DOI:
10.48550/arXiv.2304.01373.

54\. Köpf, A., et al. _OpenAssistant Conversations – Democratizing Large
Language Model Alignment_. 2023. arXiv:2304.07327 DOI:
10.48550/arXiv.2304.07327.

55\. Hu, E.J., et al. _LoRA: Low-Rank Adaptation of Large Language Models_.
2021. arXiv:2106.09685 DOI: 10.48550/arXiv.2106.09685.

56\. Lester, B., R. Al-Rfou, and N. Constant _The Power of Scale for
Parameter-Efficient Prompt Tuning_. 2021. arXiv:2104.08691 DOI:
10.48550/arXiv.2104.08691.

57\. Li, X.L. and P. Liang. _Prefix-Tuning: Optimizing Continuous Prompts for
Generation_. 2021. Online: Association for Computational Linguistics.

58\. Liu, X., et al. _P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-
tuning Universally Across Scales and Tasks_. 2021. arXiv:2110.07602 DOI:
10.48550/arXiv.2110.07602.

59\. Moor, M., et al., _Foundation models for generalist medical artificial
intelligence._ Nature, 2023. 616(7956): p. 259-265.

60\. Acosta, J.N., et al., _Multimodal biomedical AI._ Nat Med, 2022\. 28(9):
p. 1773-1784.


# Concept and Attribute Reduction Based on Rectangle Theory of Formal Concept
111The authors are supported by NSF grant of Anhui Province(No.1808085MF178),
China.

Jianqin Zhou1 , Sichun Yang1, Xifeng Wang1 and Wanquan Liu2 222Corresponding
author. Email<EMAIL_ADDRESS>

1Department of Computer Science,
Anhui University of Technology, Ma’anshan 243002, China

2School of Intelligent Systems Engineering,
Sun Yat-sen University, Shenzhen 518000, China

> Based on rectangle theory of formal concept and set covering theory, the
> concept reduction preserving binary relations is investigated in this paper.
> It is known that there are three types of formal concepts: core concepts,
> relative necessary concepts and unnecessary concepts. First, we present the
> new judgment results for relative necessary concepts and unnecessary
> concepts. Second, we derive the bounds for both the maximum number of
> relative necessary concepts and the maximum number of unnecessary concepts
> and it is a difficult problem as either in concept reduction preserving
> binary relations or attribute reduction of decision formal contexts, the
> computation of formal contexts from formal concepts is a challenging
> problem. Third, based on rectangle theory of formal concept, a fast
> algorithm for reducing attributes while preserving the extensions for a set
> of formal concepts is proposed using the extension bit-array technique,
> which allows multiple context cells to be processed by a single 32-bit or
> 64-bit operator. Technically, the new algorithm could store both formal
> context and extent of a concept as bit-arrays, and we can use bit-operations
> to process set operations ”or” as well as ”and”. One more merit is that the
> new algorithm does not need to consider other concepts in the concept
> lattice, thus the algorithm is explicit to understand and fast. Experiments
> demonstrate that the new algorithm is effective in the computation of
> attribute reductions.
>
> Keywords: Formal Concept Analysis; Concept lattice; Concept reduction;
> Attribute reduction
>
> Mathematics Subject Classification 2010: 68T30, 68T35

##  1 Introduction

Concept lattice is a key tool for information analysis and processing. The
mathematical basis of concept lattice is the lattice theory, the visualization
tool is the Hasse graph, and the research methods are abstract algebra,
discrete mathematics, data structure and algorithm analysis, fuzzy set [28],
rough set [17], granular computing [29], etc. So far, formal concept analysis
has been widely used in information retrieval [12, 4], knowledge discovery
[13], association analysis [24], recommendation system [32] and other fields
[18, 16, 23, 25].

When we decide to study a certain kind of concepts, we need to first consider
how to find out all the concepts from the specific data. This problem is
called concept lattice construction [2, 14, 1, 15, 20, 21]. Second, in order
to better analyze data and save storage space, it is necessary to reduce
concept lattice [3, 11, 8, 22, 26, 30]. Furthermore, the nodes of concept
lattice can infer from each other, and based on this, one can extract rules
[12, 9, 10, 11, 19].

Based factorization and attribute reduction, the concept reduction while
preserving binary relations is proposed in [3, 26]. The significance of a
concept in a given formal context may be different and they play different
roles in the concept reduction. These concepts can be classified into three
types: the core concepts, the relatively necessary concepts, and the
unnecessary concepts. The judgment theorems for three types of concepts have
been investigated. However, the judgment theorems for relative necessary
concepts and unnecessary concepts in [3, 26] are very complicated, and
difficult to implemented.

One can visualise a formal concept in a table of $0$ and $1$ as a closed
rectangle of $1$s, where the rows and columns are not necessarily contiguous
[1]. Based on rectangle theory of formal concept and set covering theory, we
will present the new judgment theorems for relative necessary concepts and
unnecessary concepts in this paper. Also, we deduce the bounds for both the
maximum number of relative necessary concepts and the maximum number of
unnecessary concepts. From our results, it is known that both relatively
necessary concepts and unnecessary concepts are exponential time problems.
However, by using the proposed judgment theorems, it is easier to develop a
program for computing both relative necessary concepts and unnecessary
concepts than that of [26].

Based on the discernibility matrix and Boolean function, a method to attribute
reduction is proposed while preserving the extensions in concept lattices [30,
31]. Similarly, two kinds of attribute reduction are proposed in the decision
formal context based on the maximal rules [12]. Note that the attribute
reduction in both [30, 31] and [12] can be simplified to reducing attributes
while preserving the extensions for a set of formal concepts. A fast algorithm
is proposed in this paper for reducing attributes while preserving the
extensions for a set of formal concepts. As the new algorithm store both
formal context and extension of a concept as a bit-array, and we can use bit-
operations to process set operation ”or” as well as ”and”. Further more, there
is no need to consider other formal concepts, thus the algorithm is very fast.

Technically, suppose that one column of context or the extension of a concept
is $(1,1,1,1$, $1,1,1,1$, $0,0,0,0,0,0,0,0,$ $0,0,0,0,$ $0,0,0,0$, $0,0,0,0$,
$0,0,0,0)$, it is stored as $255$ in our algorithm, where the first bit means
$2^{0}$, the second bit means $2^{1}$, the third bit means $2^{2}$ and so on.
Applying our new algorithm to mushroom data from [6], 19 columns of formal
context can be reduced while preserving the extensions for 512 formal
concepts. Experiments demonstrate that the new algorithm is much efficient in
the computation of attribute reductions.

The remainder of this paper is organized as follows. In Section 2, some basic
definitions and lemmas are reviewed. In Section 3, concept reduction
preserving binary relations is discussed, we present the new judgment theorems
for relative necessary concepts and unnecessary concepts. In Section 4, based
on rectangle theory of formal concept, a fast algorithm for reducing
attributes while preserving the extensions for a set of formal concepts is
proposed by using extension bit-array technique. Finally, in Section 5, we
conclude the paper with a summary and an outlook for future work.

##  2 Basic notions and properties

For the convenience of discussion, some basic notions and properties of formal
concepts related to this paper will be reviewed. We first present the formal
context and its operators as follows.

Definition 1. [5] A triplet $(U,A,I)$ is called a formal context, where
$U=\\{x_{1},x_{2},\ldots,x_{m}\\}$, $A=\\{a_{1},a_{2},\ldots,a_{n}\\}$, and
$I\subseteq U\times A$ is a binary relation between $U$ and $A$. Here each
$x_{i}(i\leq m)$ is called an object, and each $a_{j}(j\leq n)$ is called an
attribute. $xIa$ or $(x,a)\in I$ indicates that an object $x\in U$ has the
attribute $a\in A$.

Definition 2. [5] Let $(U,A,I)$ be a formal context. For any $X\subseteq U$
and $B\subseteq A$, two operations are defined respectively:

${}^{*}:P(U)\rightarrow P(A)$, $X^{*}=\\{m\in A|\forall g\in X,(g,m)\in I\\}$

${}^{*}:P(A)\rightarrow P(U)$, $B^{*}=\\{g\in U|\forall m\in B,(g,m)\in I\\}$

The following are the definitions of formal concepts and concept lattices.

Definition 3. [5] Let $(U,A,I)$ be a formal context. For any $X\subseteq U$
and $B\subseteq A$, if $X^{*}=B$ and $B^{*}=X$, then the pair $(X,B)$ is
called a formal concept, where $X$ and $B$ are called the extension and the
intension of $(X,B)$, respectively. For concepts
$(X_{1},B_{1}),(X_{2},B_{2})$, where $X_{1},X_{2}\subseteq U$,
$B_{1},B_{2}\subseteq A$, one can define the partial order as follows:

$(X_{1},B_{1})\leq(X_{2},B_{2})\Leftrightarrow X_{1}\subseteq
X_{2}\Leftrightarrow B_{2}\subseteq B_{1}$

Furthermore, we have the following definitions:

$(X_{1},B_{1})\bigwedge(X_{2},B_{2})=(X_{1}\cap X_{2},(B_{1}\cup B_{2})^{**})$
or $(X_{1}\cap X_{2},(X_{1}\cap X_{2})^{*})$

$(X_{1},B_{1})\bigvee(X_{2},B_{2})=((X_{1}\cup X_{2})^{**},B_{1}\cap B_{2})$
or $((B_{1}\cap B_{2})^{*},B_{1}\cap B_{2})$

Thus, all formal concepts from $(U,A,I)$ form a complete lattice, and it is
defined as a concept lattice and denoted by $L(U,A,I)$.

Lemma 1. [5] For any $X_{1},X_{2},X\subseteq U$, $B_{1},B_{2},B\subseteq A$,
here $(U,A,I)$ is a formal context, the following statements hold:

(1) $X_{1}\subseteq X_{2}\Rightarrow X^{*}_{2}\subseteq X^{*}_{1}$, $\
B_{1}\subseteq B_{2}\Rightarrow B^{*}_{2}\subseteq B^{*}_{1}$;

(2) $X\subseteq X^{**}$, $\ B\subseteq B^{**}$;

(3) $X^{*}=X^{***}$, $\ B^{*}=B^{***}$;

(4) $X\subseteq B^{*}\ \Leftrightarrow\ B\subseteq X^{*}$;

(5) $(X_{1}\cup X_{2})^{*}=X^{*}_{1}\cap X^{*}_{2},\ (B_{1}\cup
B_{2})^{*}=B^{*}_{1}\cap B^{*}_{2}$;

(6) $(X_{1}\cap X_{2})^{*}\supseteq X^{*}_{1}\cup X^{*}_{2},\ (B_{1}\cap
B_{2})^{*}\supseteq B^{*}_{1}\cup B^{*}_{2}$.

A formal context is typically represented by a table of $0$ and $1$, with $1$s
meaning binary relations between objects (rows) and attributes (columns). A
simple example of a formal context is presented as follows:

Table 1: Formal context $(G,M,I)$ $G$ | $\ \ \ \ a_{1}\ \ \ \ \ $ | $\ \ \ \ a_{2}\ \ \ \ \ $ | $\ \ \ \ a_{3}\ \ \ \ \ $ | $\ \ \ \ a_{4}\ \ \ \ \ $ | $\ \ a_{5}$
---|---|---|---|---|---
1 | 0 | 1 | 1 | 0 | 0
2 | 1 | 1 | 0 | 0 | 0
3 | 1 | 0 | 0 | 0 | 0
4 | 0 | 0 | 0 | 0 | 1
5 | 0 | 0 | 0 | 1 | 1
6 | 0 | 0 | 1 | 1 | 1
7 | 1 | 1 | 1 | 0 | 0

The formal concepts in Table 1 can be calculated as given in the following
Table 2:

Table 2: Formal concepts in Table 1 |  | $C_{0}=(\\{1,2,3,4,5,6,7\\},\emptyset)$ |
---|---|---|---
$C_{4}=(\\{2,3,7\\},\\{a_{1}\\})$ | $C_{3}=(\\{1,2,7\\},\\{a_{2}\\})$ | $C_{2}=(\\{1,6,7\\},\\{a_{3}\\})$ | $C_{1}=(\\{4,5,6\\},\\{a_{5}\\})$
| $C_{5}=(\\{2,7\\},\\{a_{1},a_{2}\\})$ | $C_{6}=(\\{1,7\\},\\{a_{2},a_{3}\\})$ | $C_{7}=(\\{5,6\\},\\{a_{4},a_{5}\\})$
| $C_{9}=(\\{7\\},\\{a_{1},a_{2},a_{3}\\})$ |  | $C_{8}=(\\{6\\},\\{a_{3},a_{4},a_{5}\\})$
|  | $C_{10}=(\emptyset,\\{a_{1},a_{2},a_{3},a_{4},a_{5}\\})$ |

In fact, one can visualise a formal concept in a table of $0$ and $1$ as a
closed rectangle of $1$s, where the rows and columns are not necessarily
contiguous [1]. Suppose we define the cell of the $i$th row and $j$th column
as $(i,j)$. Thus in Table 1, $(5,4)$, $(5,5)$, $(6,4)$ and $(6,5)$ form the
concept $C_{7}$, and $C_{7}$ is a rectangle of height $2$ and width $2$.
Similarly $(6,3)$, $(6,4)$ and $(6,5)$ form the concept $C_{8}$, and $C_{8}$
is a rectangle of height $1$ and width $3$. $(1,3)$, $(6,3)$ and $(7,3)$ form
the concept $C_{2}$, and $C_{2}$ is a rectangle of height $3$ and width $1$,
here $(1,3)$ and $(6,3)$ are not contiguous.

On the other hand, a formal concept can be obtained by applying the ∗ operator
to a set of attributes to get its extension, and then applying the ∗ operator
to the extension to get the intension. From the context in Table 1,
$\\{a_{4}\\}^{*}=\\{5,6\\}$ and $\\{5,6\\}^{*}=\\{a_{4},a_{5}\\}$. So
$(\\{5,6\\},\\{a_{4},a_{5}\\})$ is concept $C_{7}$ in Table 2. From the
rectangle theory of a formal concept, $C_{7}$ is the biggest rectangle to
cover $(5,4)$, $(5,5)$, $(6,4)$ and $(6,5)$ simultaneously.

Based on the rectangle theory of a formal concept, next we will discuss the
concept reduction preserving binary relations.

##  3 Concept reduction preserving binary relations

We first present the definition of concept reduction preserving binary
relations as follows.

Definition 4. [3, 26] Let $(G,M,I)$ be a formal context. $\mathcal{F}\subseteq
L(G,M,I)$ is called a consistent concept set preserving binary relations if

$I=\bigcup\limits_{(A_{i},B_{i})\in\mathcal{F}}A_{i}\times B_{i}$

Further, $\mathcal{F}\subseteq L(G,M,I)$ is called a concept reduction set
preserving binary relations if for any $(A,B)\in\mathcal{F}$

$I\neq\bigcup\limits_{(A_{i},B_{i})\in\mathcal{F}\backslash\\{(A,B)\\}}A_{i}\times
B_{i}$

It follows from Definitions 4 that there exists at least one concept reduction
set preserving binary relations from $L(G,M,I)$ [3].

It is obvious that

$\mathcal{O}(G,M,I)=\\{(g^{**},g^{*})|g\in G\\}$

is a consistent concept set preserving binary relations.

$\mathcal{A}(G,M,I)=\\{(m^{*},m^{**})|m\in M\\}$

is also a consistent concept set preserving binary relations [3].

For example, in Table 1, $\\{c_{1},c_{2},c_{3},c_{4},c_{7}\\}$ is a concept
reduction set preserving binary relations for formal concepts in Table 1. In
other words, $\\{c_{1},c_{2},c_{3},c_{4},c_{7}\\}$ covers exactly binary
relations in Table 1. $\\{c_{1},c_{4},c_{5},c_{6},c_{7},c_{8}\\}$ is also a
concept reduction set preserving binary relations, thus the concept reduction
set is not unique.

Based on Definition 4, the significance of a concept in a given formal context
may be different and they play different roles in the concept reduction. These
concepts can be classified into three types: the core concepts, the relatively
necessary concepts, and the unnecessary concepts. Next we provide the
characteristics of these types of concepts.

Definition 5. [3, 26] Let $(G,M,I)$ be a formal context.
$\mathcal{F}_{i}\subseteq L(G,M,I)$, where $i\in\tau$, are all concept
reductions of $L(G,M,I)$, then the concepts in $L(G,M,I)$ are classified into
three types:

(1) Core concept set: $L=\bigcap\limits_{i\in\tau}\mathcal{F}_{i}$

(2) Relatively necessary concept
set:$M=\bigcup\limits_{i\in\tau}\mathcal{F}_{i}-\bigcap\limits_{i\in\tau}\mathcal{F}_{i}$

(3) Unnecessary concept set:
$N=L(G,M,I)-\bigcup\limits_{i\in\tau}\mathcal{F}_{i}$

For example, in Table 1, $(5,4)$ is only covered by concept $c_{7}$, so
$c_{7}$ is a core concept. It is easy to show that $c_{1}$ and $c_{4}$ are
also core concepts. Thus, $c_{2}$, $c_{3}$, $c_{5}$, $c_{6}$ and $c_{8}$ are
relatively necessary concepts. Later we will explain why $c_{9}$ is an
unnecessary concept.

Based on the rectangle theory of formal concept, the following judgment
theorem of the core concept is given.

Theorem 1. [3, 26] Let $(G,M,I)$ be a formal context. For concept $(A,B)\in
L(G,M,I)$, if $(g,m)\in(A,B)$ is only covered by $(A,B)$, then concept $(A,B)$
is a core concept.

In what follows, our focus is the discussions of relatively necessary concept
and unnecessary concept. First we present the following lemma.

Lemma 2. Let $(G,M,I)$ be a formal context. For concept $(A,B)\in L(G,M,I)$,
if there exists a set $A^{\prime}\subseteq G\backslash A$ such that
$B=\bigcup\limits_{g\in A^{\prime}}g^{*}$, or there exists a set
$B^{\prime}\subseteq M\backslash B$ such that $A=\bigcup\limits_{m\in
B^{\prime}}m^{*}$, then $(A,B)$ is an unnecessary concept, and $(A,B)$ is
called a side covered concept.

###### Proof.

Without loss of generality, we assume that $B=\bigcup\limits_{g\in
A^{\prime}}g^{*}$.

For any $(x,y)\in(A,B)$, suppose that $y\in g^{*}$. As $g\notin A$, so
$(g,y)\notin(A,B)$.

For any $(C,D)$, such that $(g,y)\in(C,D)$, by Lemma 1.(1),

$g\in C\Rightarrow D\subseteq g^{*}\Rightarrow D\subseteq B\Rightarrow
A\subseteq C\Rightarrow(x,y)\in(C,D)$

Suppose that $\mathcal{F}\subseteq L(G,M,I)$ is a consistent concept set, then
$\mathcal{F}\backslash\\{(A,B)\\}$ is still a consistent concept set. Thus
$(A,B)$ is an unnecessary concept.  $\hbox to0.0pt{$\sqcap$\hss}\sqcup$

For example, in Table 1, $\\{a_{1},a_{2},a_{3}\\}=\\{1\\}^{*}\cup\\{2\\}^{*}$,
so $C_{9}=(\\{7\\},\\{a_{1},a_{2},a_{3}\\})$ is an unnecessary concept.

If we visualise a formal concept as a rectangle, then the following lemma is
obvious.

Lemma 3. Let $(G,M,I)$ be a formal context. Concept $(A,B)\in L(G,M,I)$ must
be unnecessary if $(A,B)$ is covered by some core concepts, where cover means
for any $(g,m)\in(A,B)$, $(g,m)$ is also in a core concept.

Table 3: Formal context $(G,M,I_{3})$ $G$ | $\ \ \ \ a_{1}\ \ \ \ \ $ | $\ \ \ \ a_{2}\ \ \ \ \ $ | $\ \ \ \ a_{3}\ \ \ \ \ $ | $\ \ \ \ a_{4}\ \ \ \ \ $ | $\ \ a_{5}$
---|---|---|---|---|---
1 | 1 | 1 | 1 | 0 | 0
2 | 1 | 1 | 1 | 0 | 0
3 | 1 | 1 | 1 | 0 | 0
4 | 0 | 1 | 1 | 1 | 1
5 | 0 | 1 | 1 | 1 | 1
6 | 0 | 1 | 1 | 1 | 1

For example, in Table 3, there are concepts
$C_{1}=(\\{4,5,6\\},\\{a_{2},a_{3},a_{4},a_{5}\\})$,
$C_{2}=(\\{1,2,3\\},\\{a_{1},a_{2},a_{3}\\})$ and
$C_{3}=(\\{1,2,3,4,5,6\\},\\{a_{2},a_{3}\\})$. $(1,a_{1})$ is only covered by
$C_{2}$, so $C_{2}$ is a core concept. $(4,a_{5})$ is only covered by $C_{1}$,
so $C_{1}$ is a core concept. If we visualise a formal concept as a rectangle,
$C_{3}$ is covered by $C_{2}$ and $C_{1}$, so $C_{3}$ is an unnecessary
concept.

Based on Lemma 2 and Lemma 3, we have the following judgment theorem of
unnecessary concept.

Theorem 2. Let $(G,M,I)$ be a formal context. For $(A,B)\in L(G,M,I)$ is not a
core concept, $(A,B)$ is an unnecessary concept if and only if $(A,B)$ is
covered by some core concepts, or covered by some side covered concepts, or
covered by both some core concepts, and some side covered concepts, where
cover means for any $(g,m)\in(A,B)$, $(g,m)$ is also in a core concept, or in
a side covered concept.

###### Proof.

$\Leftarrow$ It follows immediately from Lemma 2 and Lemma 3.

$\Rightarrow$ Let

$G^{\prime}=G\backslash A,\mathcal{O}^{\prime}(G,M,I)=\\{(g^{**},g^{*})|g\in
G^{\prime}\\}$ $M^{\prime}=M\backslash
B,\mathcal{A}^{\prime}(G,M,I)=\\{(m^{*},m^{**})|m\in M^{\prime}\\}$
$\mathcal{F}=\mathcal{O}^{\prime}(G,M,I)\cup\mathcal{A}^{\prime}(G,M,I)$

$(A,B)\in L(G,M,I)$ is not a core concept, thus for each $(g,m)\in(A,B)$,
$(g,m)$ is covered by at least two concepts. Therefore, all core concepts are
in $\mathcal{F}$.

$(A,B)$ is an unnecessary concept, thus

$I=\bigcup\limits_{(A_{i},B_{i})\in\mathcal{F}}A_{i}\times B_{i}$

Therefore, $(A,B)$ is covered by some core concepts, or covered by some side
covered concepts, or covered by both some core concepts, and some side covered
concepts.  $\hbox to0.0pt{$\sqcap$\hss}\sqcup$

We also have the following judgment theorem of relatively necessary concept.

Theorem 3. Let $(G,M,I)$ be a formal context. For $(A,B)\in L(G,M,I)$ is not a
core concept, $(A,B)$ is a relatively necessary concept if and only if

$I\neq\bigcup\limits_{(A_{i},B_{i})\in\mathcal{F}}A_{i}\times B_{i}$

where

$G^{\prime}=G\backslash A,\mathcal{O}^{\prime}(G,M,I)=\\{(g^{**},g^{*})|g\in
G^{\prime}\\}$ $M^{\prime}=M\backslash
B,\mathcal{A}^{\prime}(G,M,I)=\\{(m^{*},m^{**})|m\in M^{\prime}\\}$
$\mathcal{F}=\mathcal{O}^{\prime}(G,M,I)\cup\mathcal{A}^{\prime}(G,M,I)$

It should be noted that with our simple judgment theorems, it is easy to
develop a program for computing both relative necessary concepts and
unnecessary concepts.

Table 4: Formal context $(G,M,I_{4})$ $G$ | $\ \ \ \ a_{1}\ \ \ \ \ $ | $\ \ \ \ a_{2}\ \ \ \ \ $ | $\ \ \ \ a_{3}\ \ \ \ \ $ | $\ \ \ \ a_{4}\ \ \ \ \ $ | $\ \ a_{5}$
---|---|---|---|---|---
1 | 1 | 1 | 1 | 0 | 0
2 | 1 | 1 | 1 | 0 | 0
3 | 1 | 1 | 1 | 0 | 0
4 | 0 | 1 | 1 | 1 | 1
5 | 0 | 1 | 1 | 1 | 1
6 | 0 | 1 | 1 | 1 | 1
7 | 1 | 1 | 0 | 0 | 0
8 | 1 | 0 | 1 | 0 | 0

For example, in Table 4, there are concepts
$C_{1}=(\\{4,5,6\\},\\{a_{2},a_{3},a_{4},a_{5}\\})$,
$C_{2}=(\\{1,2,3\\},\\{a_{1},a_{2},a_{3}\\})$ and
$C_{3}=(\\{1,2,3,4,5,6\\},\\{a_{2},a_{3}\\})$.
$\\{a_{1},a_{2},a_{3}\\}=\\{7\\}^{*}\cup\\{8\\}^{*}$, so
$C_{2}=(\\{1,2,3\\},\\{a_{1},a_{2},a_{3}\\})$ is a side covered concept.
$(4,a_{5})$ is only covered by $C_{1}$, so $C_{1}$ is a core concept. If we
visualise a formal concept as a rectangle, $C_{3}$ is covered by $C_{2}$ and
$C_{1}$, so $C_{3}$ is an unnecessary concept.

In Table 4, consider concept $C_{4}=(\\{1,2,3,7\\},\\{a_{1},a_{2}\\})$. It is
easy to show that $C_{4}$ is not a core concept. Further,
$\mathcal{O}^{\prime}(G,M,I)=\\{(g^{**},g^{*})|g\in
G^{\prime}\\}=\\{c_{1},c_{5}\\}$, where
$C_{1}=(\\{4,5,6\\},\\{a_{2},a_{3},a_{4},a_{5}\\})$,
$C_{5}=(\\{1,2,3,8\\},\\{a_{1},a_{3}\\})$.
$\mathcal{A}^{\prime}(G,M,I)=\\{(m^{*},m^{**})|m\in
M^{\prime}\\}=\\{c_{1},c_{6}\\}$, where
$C_{6}=(\\{1,2,3,4,5,6,8\\},\\{a_{3}\\})$. As

$\mathcal{F}=\mathcal{O}^{\prime}(G,M,I)\cup\mathcal{A}^{\prime}(G,M,I)=\\{c_{1},c_{5},c_{6}\\}$

$C_{4}$ is not covered by $\mathcal{F}$, thus $C_{4}$ is a relatively
necessary concept.

To further discuss the characteristics of concepts of each type, we consider
the maximum number of concepts of each type. As a core concept is in
$\mathcal{O}(G,M,I)\cap\mathcal{A}(G,M,I)$, the following lemma is
straightforward.

Lemma 4. Let $(G,M,I)$ be a formal context, and $m=|G|$, $n=|M|$. Then the
maximum number of core concepts is not greater than $\min(m,n)$.

Table 5: Formal context $(G,M,I_{5})$ $G$ | $\ \ \ \ a_{1}\ \ \ \ \ $ | $\ \ \ \ a_{2}\ \ \ \ \ $ | $\ \ \ \ a_{3}\ \ \ \ \ $ | $\ \ \ \ a_{4}\ \ \ \ \ $ | $\ \ a_{5}$
---|---|---|---|---|---
1 | 1 | 1 | 0 | 0 | 0
2 | 1 | 0 | 1 | 0 | 0
3 | 1 | 0 | 0 | 1 | 0
4 | 1 | 0 | 0 | 0 | 1
5 | 0 | 1 | 1 | 0 | 0
6 | 0 | 1 | 0 | 1 | 0
7 | 0 | 1 | 0 | 0 | 1
8 | 0 | 0 | 1 | 1 | 0
9 | 0 | 0 | 1 | 0 | 1
10 | 0 | 0 | 0 | 1 | 1

For example, in Table 7, there are 5 concepts in $\mathcal{A}(G,M,I)$, and
there are 10 concepts in $\mathcal{O}(G,M,I)$, but there is neither core
concepts nor unnecessary concepts. The number of relatively necessary concepts
is

$10+5=\binom{5}{2}+5=\binom{|M|}{\lfloor|M|/2\rfloor}+|M|$

Inspired by this example, we have the following lemma.

Lemma 5. Let $(G,M,I)$ be a formal context, and $n=|M|$. Suppose
$|G|>\binom{n}{\lfloor n/2\rfloor}$, then the maximum number of relatively
necessary concepts is not less than

$\binom{n}{\lfloor n/2\rfloor}+n$

where $\lfloor x\rfloor$ is the largest number that is less than or equal to
$x$.

###### Proof.

Note that $\binom{n}{k}=\binom{n}{n-k}$, and $\binom{n}{\lfloor n/2\rfloor}$
is the maximum, where $1\leq k\leq n$.

For the formal context $(G,M,I)$, let each object (corresponding to a row in
the table) have $\lfloor n/2\rfloor$ attributes. According to combinatorial
mathematics, there are $\binom{n}{\lfloor n/2\rfloor}$ different objects. The
$k$th object forms a concept $(\\{k\\}^{**},\\{k\\}^{*})$, where
$\\{k\\}^{**}=\\{k\\}$, which is not covered by

$\mathcal{F}=\mathcal{O}^{\prime}(G,M,I)\cup\mathcal{A}^{\prime}(G,M,I)$

where

$\mathcal{O}^{\prime}(G,M,I)=\\{(g^{**},g^{*})|g\in G\backslash\\{k\\}\\}$
$\mathcal{A}^{\prime}(G,M,I)=\\{(m^{*},m^{**})|m\in M\backslash\\{k\\}^{*}\\}$

Thus $(\\{k\\}^{**},\\{k\\}^{*})$ of the $k$th object is relatively necessary.

In the table of formal context $(G,M,I)$, if there exist two same columns,
which means the values of the two positions are always the same. This is a
contradiction by combinatorial mathematics, thus there are $n$ different
columns. Similarly, one can prove that $(\\{j\\}^{*},\\{j\\}^{**})$, where
$\\{j\\}^{**}=\\{j\\}$ and $1\leq j\leq n$, of the $j$th attribute is
relatively necessary.

Therefore, there are $\binom{n}{\lfloor n/2\rfloor}+n$ relatively necessary
concepts.

$\hbox to0.0pt{$\sqcap$\hss}\sqcup$

Table 6: Formal context $(G,M,I_{6})$ $G$ | $\ \ \ \ a_{1}\ \ \ \ \ $ | $\ \ \ \ a_{2}\ \ \ \ \ $ | $\ \ \ \ a_{3}\ \ \ \ \ $ | $\ \ \ \ \ a_{4}\ \ \ \ $
---|---|---|---|---
1 | 1 | 1 | 0 | 0
2 | 1 | 0 | 1 | 0
3 | 1 | 0 | 0 | 1
4 | 0 | 1 | 1 | 0
5 | 0 | 1 | 0 | 1
6 | 0 | 0 | 1 | 1
7 | 1 | 1 | 1 | 0
8 | 1 | 1 | 0 | 1
9 | 1 | 0 | 1 | 1
10 | 0 | 1 | 1 | 1

In Table 6, let $C_{1}=(\\{1,7\\},\\{a_{1},a_{2}\\})$,
$C_{2}=(\\{2,7\\},\\{a_{1},a_{3}\\})$,
$C_{7}=(\\{7\\},\\{a_{1},a_{2},a_{3}\\})$, then $C_{7}$ is covered by $C_{1}$
and $C_{2}$, thus $C_{7}$ is an unnecessary concept. In fact, if we add one
more object with 3 attributes, then this object (or row) must form an
unnecessary concept. Thus we have the following lemma.

Lemma 6. Let $(G,M,I)$ be a formal context, and $n=|M|$. Suppose
$|G|>\binom{n}{\lfloor n/2\rfloor}+\binom{n}{\lfloor n/2+1\rfloor}$, then the
maximum number of unnecessary concepts is not less than

$\binom{n}{\lfloor n/2+1\rfloor}$

where $\lfloor x\rfloor$ is the largest number that is less than or equal to
$x$.

From the discussion above, we know that both relatively necessary concepts and
unnecessary concepts are exponential time problems.

##  4 An algorithm for attribute reduction

Based on the discernibility matrix and Boolean function, a method of attribute
reduction is proposed while preserving the extensions in concept lattices in
[30, 31]. Similarly, two kinds of attribute reduction are proposed in the
decision formal context based on the maximal rules [12], and an approach to
computing attribute reductions is proposed based on the discernibility matrix
and Boolean function. However, the approaches in [30, 31, 12] are very
difficult to implemented.

Note that the attribute reduction in both [30, 31] and [12] can be simplified
to reducing attributes while preserving the extensions for a set of formal
concepts. We also know that for any formal concept $(X,B)$ of $L(U,A,I)$,
$X^{*}=B$ and $B^{*}=X$. Based on this explicit idea, a fast algorithm for
reducing attributes while preserving the extensions for a set of formal
concepts is proposed in this paper. Technically, by checking $X^{**}=X$ to
determine whether to delete a column of formal context or not.

As the new algorithm would store both formal context and extension of a
concept as a bit-array, and we can use bit-operations to process set
operations ”or” as well as ”and”. What is more, there is no need to consider
other formal concepts in $L(U,A,I)$, thus the algorithm, given below with
analysis, is very fast.

Line 2 – Construct an intension of $X$, which could be different from $B$. It
is implemented in C language as the following.

memset(Row, 0, sizeof(Row)); // clear the Row, which is used to store the
intension

for(q=0; q<n; q++) $\\{$

for(int k = (m - 1)/32; k >= 0; k- -) $\\{$

// check whether the qth column contains the ith extension column

if((contextCol[k][q] & extensionCol[k][i]) != extensionCol[k][i])break;

$\\}$

// the qth column contains the ith extension column, so to change the
intension

if(k<0) Row[q>>5] $|=$ (1<<(q%32));

$\\}$

Note that here $m=|U|,n=|A|$, and both formal context contextCol and extension
of a concept extensionCol are stored as bit-arrays (in the form of 32-bit
unsigned integers). Without processing each object in the formal context and
extension of a concept, the bit and operation is used when check whether the
$q$th column of formal context contains the $i$th extension column.

First call the procedure to check whether to reduce the first column. If
return true, then let the first column of the formal context $(U,A,I)$ be
zero, otherwise there is no change. Second call the procedure to check whether
to reduce the second column. If return true, then let the second column of the
formal context $(U,A,I)$ be zero, otherwise there is no change. Keep this
process going and at the end of this process, we get an attribute reduction
preserving the extensions for a set of formal concepts.

Suppose that first call the procedure to check whether to reduce the second
column, second call the procedure to check whether to reduce the third column.
Finally call the procedure to check whether to reduce the first column. At the
end of this process, we may get a different attribute reduction preserving the
same set of formal concepts.

Thus by calling the procedure with different starting $i$th column, where
$1\leq i\leq n$, we can get all attribute reductions preserving the same set
of formal concepts.

Some experiments are done to compute attribute reductions with the new
algorithm. The experiment results are given in Table 7. Here mushroom data and
nursery data are from [6]. The experiments are carried out using a laptop
computer with an Intel Core i5-2450M 2.50 GHz processor and 8GB of RAM.

Table 7: Computation of attribute reductions with the new algorithm | Mushroom | Nursery |
---|---|---|---
$|G|\times|M|$ | $8,124\times 115$ | $12,960\times 30$ |
Number of concepts | 512 | 512 |
Time in seconds | 4.218 | 0.582 |
Number of reduced columns | 19 | 2 |

In Table 7, In-Close4 algorithm from [2] is used to compute all the formal
concepts for mushroom data. From which, we take the first 512 concepts. With
our new algorithm, 19 columns of formal context can be reduced while
preserving the extensions for 512 formal concepts. Experiments demonstrate
that the new algorithm is much effective in the computation of attribute
reductions.

##  5 Conclusions and future work

A pair $(X,B)$ is called a formal concept if $X^{*}=B$ and $B^{*}=X$, which is
a frequently used definition. With another perspective, based on rectangle
theory of formal concept and set covering theory, the concept reduction
preserving binary relations was discussed. It is known that there are three
types of formal concepts: core concepts, relative necessary concepts and
unnecessary concepts. We presented the new judgment theorems for relative
necessary concepts and unnecessary concepts. Also, we gave bounds for both the
maximum number of relative necessary concepts and the maximum number of
unnecessary concepts. From our results, it is known that both relatively
necessary concepts and unnecessary concepts are exponential time problems.

As the attribute reduction approaches in both [30] and [12] can be simplified
to reducing attributes while preserving the extensions for a set of formal
concepts, a fast algorithm was proposed for reducing attributes while
preserving the extensions for a set of formal concepts. Experiments
demonstrated that the new algorithm is much effective in the computation of
attribute reductions.

Object oriented concept lattice is a more extensive concept lattice [27]. So,
it is more difficult to reduce concepts or attributes in object oriented
concept lattices. In the future, we will apply the data structure and
technique in these algorithms to object oriented concept lattices, attribute
oriented concept lattices and so on.

## References

  * [1] S. Andrews, A ‘Best-of-Breed’ approach for designing a fast algorithm for computing fixpoints of Galois Connections, Information Sciences, 295 (20) (2015) 633–649.
  * [2] S. Andrews, In-Close4 Program, 2017, https://sourceforge.net/projects/inclose/files/In-Close/.
  * [3] L. Cao, L. Wei, J. J. Qi. Concept reduction preserving binary relations (in Chinese). Pattern Recogn Artif Intell, 2018, 31(6): 516-524
  * [4] C. Carpineto, G. Romano, A Lattice Conceptual Clustering System and Its Application to Browsing Retrieval. Machine Learning, 1996, 24(2): 95-122.
  * [5] B. Ganter, R. Wille, Formal Concept Analysis: Mathematical Foundations, Springer-Verlag, New York, 1999.
  * [6] A. Frank, A. Asuncion, UCI Machine Learning Repository, 2010, http://archive.ics.uci.edu/ml.
  * [7] S.O. Kuznetsov, On computing the size of a lattice and related decision problems, Order, 18 (4) (2001) 313-321
  * [8] S.O. Kuznetsov, S.A. Obiedkov, C. Roth, Reducing the representation complexity of lattice-based taxonomies, in: Conceptual Structures: Knowledge Architectures for Smart Applications, Proceedings of the 15th International Conference on Conceptual Structures, ICCS 2007, Sheffield, UK, July 22-27, 2007, pp.241–254.
  * [9] J. Li, C. Mei, Y. Lv, Incomplete decision contexts: approximate concept construction, rule acquisition and knowledge reduction, Int. J. Approx. Reason. 54(1) (2013) 149–165.
  * [10] J. Li, C. Mei, L. Wang, J. Wang, On inference rules in decision formal contexts,  International Journal of Computational Intelligence Systems. 8(1) (2015) 175–186.
  * [11] J. Li, C. Mei, J. Wang, X. Zhang, Rule-preserved object compression in formal decision contexts using concept lattices, Knowl.-Based Syst. 71 (2014) 435–445.
  * [12] L. Li, J. Mia, B. Xie, Attribute reduction based on maximal rules in decision formal context,  International Journal of Computational Intelligence Systems. 7(6)(2014) 1044-1053.
  * [13] P. H. P. Nguyen, D. Corbett, A Basic Mathematical Framework for Conceptual Graphs, IEEE Transactions on Knowledge and Data Engineering, 2006, 18(2): 261-271.
  * [14] J. Outrata, V. Vychodil, Fast algorithm for computing fixpoints of Galois connections induced by object-attribute relational data, Information Sciences, 185 (1) (2012) 114-127
  * [15] P. Osicka, Algorithms for computation of concept trilattice of triadic fuzzy context, in: Advances in Computational Intelligence -Proceedings of the 14th International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems, IPMU 2012, Catania, Italy, July 9-13, 2012, pp.221-230 (Part III).
  * [16] N. Pasquier, Y. Bastide, R. Taouil, L. Lakhal, Efficient mining of association rules using closed itemset lattices, Inf. Syst. 24(1) (1999) 25–46.
  * [17] Z. Pawlak, Rough Sets, International Journal of Computer and Information Sciences, 1982, 11(5): 341-356.
  * [18] J. Poelmans, D.I. Ignatov, S. Viaene, G. Dedene, S.O. Kuznetsov, Text mining scientific papers: a survey on FCA-based information retrieval research, in:Advances in Data Mining. Applications and Theoretical Aspects -Proceedings of the 12th Industrial Conference, ICDM 2012, Berlin, Germany, July 13-20, 2012, pp.273-287.
  * [19] Z. Pei, D. Ruan, D. Meng, Z. Liu, Formal concept analysis based on the topology for attributes of a formal context, Information Sciences, 236 (2013) 66–82.
  * [20] J. Qi, W. Liu, L. Wei, Computing the set of concepts through the composition and decomposition of formal contexts, in: International Conference on Machine Learning and Cybernetics, Proceedings, ICMLC 2012, Xian, Shaanxi, China, July 15–17, 2012, pp.1326–1332.
  * [21] J. Qi, T. Qian, L. Wei, The connections between three-way and classical concept lattices, Knowl.-Based Syst. 91 (2016) 143–151.
  * [22] R. Ren, L. Wei, The attribute reductions of three-way concept lattices, Knowl.-Based Syst. 99 (2016) 92–102.
  * [23] M. Shao, H. Yang, W. Wu, Knowledge reduction in formal fuzzy contexts, Knowl.-Based Syst. 73 (2015) 265–275.
  * [24] X. D. Tu, Y. L. Wang, M. L. Zhang, et al. Using Formal Concept Analysis to Identify Negative Correlations in Gene Expression Data, IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2016, 13(2): 380-391.
  * [25] Q. Wan, L. Wei, Approximate concepts acquisition based on formal contexts, Knowl.-Based Syst. 75 (2015) 78–86.
  * [26] L. Wei, L. Cao, J. J. Qi , et al. Concept reduction and concept characteristics in formal concept analysis (in Chinese). Sci Sin Inform, 2020, 50(12): 1817–1833.
  * [27] Y. Y. Yao, Concept lattices in rough set theory, Processing Nafips 04 IEEE Meeting of the the Fuzzy Information, Canada: IEEE, September 27,2004: 796-801.
  * [28] L. A. Zadeh, Fuzzy Sets. Information and Control, 1965, 8(3): 338-353.
  * [29] L. A. Zadeh, Toward a Theory of Fuzzy Information Granulation and Its Centrality in Human Reasoning and Fuzzy Logic. Fuzzy Sets and Systems, 1997, 90(2): 111-127.
  * [30] W. X. Zhang, L. Wei and J. J. Qi, Attribute reduction in concept lattice based on discernibility matrix,  Lecture Notes in Computer Science, 3642, 157-165 (2005).
  * [31] W. X. Zhang, L. Wei and J. J. Qi, Attribute reduction theory and approach to concept lattice, Sci China Ser F-Inf Sci, 2005, 35(6): 628-639
  * [32] C. F. Zou, D. Q. Zhang, J. F. Wan, et al. Using Concept Lattice for Personalized Recommendation System Design, IEEE Systems Journal, 2017, 11(1): 305-314.