"Proposes using text-based negotiation games as benchmarks for evaluating LLMs’ reasoning and decision making capabilities, by using Chain-of-Thought prompts (CoT)." [gal30b+] #CL #CY #LG

https://github.com/S-Abdelnabi/LLM-Deliberation
https://arxiv.org/abs/2309.17234v1 #arxiv

Figure 18: Examples of GPT-3.5 models (with the best prompt inTable 1). Agents often perform wrong mappings of sub-options to the corresponding scores (a), wrong calculations of scores (a), wrong comparisons to their minimum thresholds (b), and wrong inferences about other parties (a). They also often reveal scores in their final answer (b) and do not perform adequate exploration of feasible solutions.

Figure 12: Example of the final deal proposed by p1 in one greedy game. A consensus on issues raised by the greedy agent can lead to less favorable decisions w.r.t. p1 itself; cooperative agents may over-compromise; this might eventually lead to no agreement if p1’s score is not met. In the game rules given to p1, if all parties agree, it will receive an additional score of 10.

Figure 3: Example from a negotiation session. The agent takes the previous interactions appended to its initial prompts. The prompt incentivized the agent to cooperate and is structured as observation, exploration, and planning steps.

Figure 1: Left: 6 parties negotiate over 5 issues (A, B, ..., E) with different sub-options (b1, b2, etc.). The maximum score any party can achieve is 100. Each party has its own secret scores for the sub-options and a minimum threshold for acceptance. The pie charts represent the priority of issues and the most preferred sub-option. Right: A depiction of how parties can compromise to reach a common agreement that increases their collective average score by finding adjustments to their ideal deal. The graph is the result of one of our experiments with GPT-4. Over rounds, the leading agent p1 proposes deals in its turn that reduce its own score (while still being above its own minimum threshold) but increase the average collective score of all agents (which p1 cannot directly observe).

**arXiv Comp. Linguistics** @arxiv_cl@creative.ai · Sep 29, 2023

Sep 29, 2023

arXiv Comp. Linguistics @arxiv_cl@creative.ai

How We Define Harm Impacts Data Annotations: Explaining How Annotators Distinguish Hateful, Offensive, and Toxic Comments

"Uses Venn Diagrams, information gain comparison, and content analysis to study the differences between annotator's use of three concepts - hateful, offensive, and toxic and to identify the factors that explain these differences." [gal30b+] #CL #CY

https://arxiv.org/abs/2309.15827v1 #arxiv

Figure 1. Venn diagram showing overlap among H, O, and T.

**arXiv Comp. Linguistics** @arxiv_cl@creative.ai · Sep 27, 2023

Sep 27, 2023

arXiv Comp. Linguistics @arxiv_cl@creative.ai

LLMCarbon: Modeling the End-to-End Carbon Footprint of Large Language Models

"A model to compute a carbon footprint estimate given architectural parameters, hardware usage, and training hyperparameters, for dense and mixture-of-experts (MoE) LLMs." [gal30b+] #CL #AI #CY #LG

https://arxiv.org/abs/2309.14393v1 #arxiv

Figure 6: The carbon footprint of three LLMs in case studies.

Figure 7: The carbon footprint of GPT3 trained by different computing devices.

Figure 8: The trade-off between training carbon footprint and test loss.

**arXiv Comp. Linguistics** @arxiv_cl@creative.ai · Sep 24, 2023

Sep 24, 2023

arXiv Comp. Linguistics @arxiv_cl@creative.ai

The Cambridge Law Corpus: A Corpus for Legal AI Research

"The CLC corpus consists of court cases from the UK, mostly from the last decade, as well as earlier cases going back to the 16th century." [gal30b+] #CL #CY

https://github.com/cambridge-ai-and-law-project/CambridgeLawCorpus"
https://arxiv.org/abs/2309.12269v1 #arxiv

Figure 1: A simplified view of the UK court and tribunal structure (Judiciary UK, 2023).

Figure 2: Proportion of words in documents belonging to the listed topics. A word can belong to more than one topic. Left: Aggregated to a one year period spanning 1950-2020. Centre: Aggregated to a one year period spanning 2000-2020. Right: Aggregated to a ten year period spanning 1573-2020.

Figure 6: Number of cases per court in descending order.

Figure 9: An example of the case outcome annotations in the CLC.

**arXiv Comp. Linguistics** @arxiv_cl@creative.ai · Sep 24, 2023

Sep 24, 2023

arXiv Comp. Linguistics @arxiv_cl@creative.ai

Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detection

"An adaptive rationale guidance network (ARG) for fake news detection is proposed that utilizes the knowledge of the large language model to guide the small language model in fake news detection." [gal30b+] #CL #AI #CY

https://github.com/ICTMCG/ARG
https://arxiv.org/abs/2309.12247v1 #arxiv

Figure 3: Overall architecture of our proposed adaptive rationale guidance (ARG) network and its rationalefree version ARG-D. In the ARG, the news item and LLM rationales are (a) respectively encoded into X and R∗(∗ ∈ {t, c}). Then the small and large LMs collaborate with each other via news-rationale feature interaction, LLM judgment prediction, and rationale usefulness evaluation. The obtained interactive features f ′∗→x (∗ ∈ {t, c}). These features are finally aggregated with attentively pooled news feature x for the final judgment. In the ARG-D, the news encoder and the attention module are preserved and the output of the rationale-aware feature simulator is supervised by the aggregated feature fcls for knowledge distillation.

Figure 1: Illustration of the role of large language models (LLMs) in fake news detection. In this case, (a) the LLM fails to output correct judgment of news veracity but (b) helps the small language model (SLM) judge correctly by providing informative rationales.

Figure 2: Illustration of prompting approaches for LLMs.

Figure 4: Statistics of additional correctly judged samples of (a) ARG and (b) ARG-D over the BERT baseline. right(·) denotes samples correctly judged by the method (·). TD/CS: Textual description/commonsense perspective.

**arXiv Comp. Linguistics** @arxiv_cl@creative.ai · Sep 24, 2023

Sep 24, 2023

arXiv Comp. Linguistics @arxiv_cl@creative.ai

Focal Inferential Infusion Coupled with Tractable Density Discrimination for Implicit Hate Speech Detection

"Focused Inferential Adaptive Density Discrimination, a novel framework that enhances implicit hate speech detection by bringing the surface form of an implicit hate speech closer to its implied form while increasing the inter-cluster distance among various class labels." [gal30b+] #CL #CY

https://github.com/LCS2-IIITD/FIADD
https://arxiv.org/abs/2309.11896v1 #arxiv

Fig. 4. 2D t-SNE plots of the last hidden representations obtained for the implicit class and its respective inferential (implied) set for the AbuseEval (a,b,c), ImpGab (d,e,f) and LatentHate (g,h,i) datasets. 0.18, 0.18, and 0.03 are the silhouette scores for cases (a), (b), and (c), respectively, in AbuseEval. 0.18, 0.23, and 0.07 are the silhouette scores for cases (d), (e), and (f), respectively, in ImpGab. 0.14, 0.13, and 0.01 are the silhouette scores for cases (g), (h), and (i), respectively, in LatentHate. In all cases, the lower the score, the better the system captures the closeness between the surface form and the implied form of hate.

Fig. 2. Architecture for Focused Inferential Adaptive Density Discrimination (FiADD). X: a set of texts, implied annotations (only for implicit class), and class labels, PLM: pre-trained language model (frozen), 𝑅ℎ : hidden layer representation of 128 dimensions, 𝑅𝑜𝑢𝑡 : output classification head (CH) with the size of output labels. [𝑅′ 𝑛ℎ𝑎𝑡𝑒 , 𝑅′ 𝑒𝑥𝑝 and 𝑅′ 𝑖𝑚𝑝 ]: a subset of 𝑅ℎ representing selected (seed and imposter clusters) local densities of non-hate, explicit and implicit class, respectively. 𝑅′ 𝑖𝑛𝑓 : representation for respective inferential samples for selected implicit samples. ACE is the alpha cross-entropy (CE with class weights). 𝐴𝐷𝐷𝐼𝑛𝑓 and 𝐴𝐷𝐷𝐼𝑛𝑓 +𝑓 𝑜𝑐 are the adaptive density discriminator with inferential only and inferential + focused objective, respectively.

Fig. 1. Symbolic representation of Focused Inferential Adaptive Density Discrimination (FiADD) as applied to implicit hate speech detection. It combines three distance-metric objectives: (a) adaptive density discrimination to discriminate local clusters (this could be between non-hate and implicit hate, non-hate and explicit hate, or implicit hate and explicit hate), (b) focus on penalizing samples near the discriminant boundary (a.k.a non-uniform weightage to samples in the cluster), and (c) reduce the symbolic difference between the surface and semantic form of the textual content (the distance between implicit hate and implied hate).

Fig. 3. 2D t-SNE plots of the last hidden representations after applying K-means (K=3) on the implicit class for the AbuseEval (a,b,c), ImpGab (d,e,f) and LatentHate (g,h,i) datasets. {0, 1, 2} are the random cluster ids. 0.34, 0.31, and 0.51 are the silhouette scores for cases (a), (b), and (c), respectively, in AbuseEval. 0.38, 0.24, and 0.52 are the silhouette scores for cases (d), (e), and (f), respectively, in ImpGab. 0.32, 0.29, and 0.32 are the silhouette scores for cases (d), (e), and (f), respectively, in LatentHate. In all cases, the higher the score, the better discriminated the clusters.

**arXiv Comp. Linguistics** @arxiv_cl@creative.ai · Sep 18, 2023

Sep 18, 2023

arXiv Comp. Linguistics @arxiv_cl@creative.ai

Casteist but Not Racist? Quantifying Disparities in Large Language Model Bias Between India and the West

"The LLMs tested are found to exhibit strong stereotypical and anti-stereotypical biases in the Indian caste and religion contexts, especially as compared to the Western context." [gal30b+] #CL #CY

https://github.com/khyatikhandelwal/Indian-LLMs-Bias
https://arxiv.org/abs/2309.08573v1 #arxiv

Figure 1: Encoder LLM bias scores, presented as deviation from stereo and anti-stereo parity (i.e., a bias score of 50%). The seperation by bias axis and model type shows how religion and caste bias scores are higher on an average for each of the models, while race scores are negative. The caste based bias scores are the highest overall.

**arXiv Comp. Linguistics** @arxiv_cl@creative.ai · Sep 15, 2023

Sep 15, 2023

arXiv Comp. Linguistics @arxiv_cl@creative.ai

Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics

"Visual instruction tuning, a prevailing strategy for transitioning LLMs into MLLMs, unexpectedly helps models attain both improved truthfulness and ethical alignment in the pure NLP context." [gal30b+] #CL #AI #CV #CY #LG

github.com/UCSC-VLAA/Sight-Beyond-Text
https://arxiv.org/abs/2309.07120v1 #arxiv

Figure 1: Visual instruction tuning substantially improves the truthfulness and ethics of LLMs. We observe that tuning LLMs with only 80k multi-modal data can yield stronger results on truthfulness and ethics than those with over one million human-annotated RLHF data. Note that these LLMs employ images only during the visual instruction tuning and are tested without images for NLP tasks.

Figure 2: Performance of visual-instruction-tuned LLaMA2 models and text instruction tuned ones on Ethics and TruthfulQA benchmarks. The text-only visual instruction data is taken directly from LLaVA, but without the paired images.

Figure 3: Results of different data components on Ethics and TruthfulQA of visual-instructiontuned LLMs. We utilize 20K of different forms of data (Conversation, Details, Reasoning), and additionally sample 20K data out of the original 80K training instances (Random 20K) for comparison.

**arXiv Comp. Linguistics** @arxiv_cl@creative.ai · Sep 14, 2023

Sep 14, 2023

arXiv Comp. Linguistics @arxiv_cl@creative.ai

Down the Toxicity Rabbit Hole: Investigating PaLM 2 Guardrails

"Uses a toxicity rabbit hole framework, which instructs the model to generate more toxic content than the previous iteration until it throws a safety violation alert by PaLM 2 safety guard rails." [gal30b+] #CL #CY

https://arxiv.org/abs/2309.06415v1 #arxiv

Figure 3: Distribution of toxic rabbit hole depths.

Figure 7: Average rabbit hole depth with varying temperature and top_k.

Figure 4: Distribution of content criticism, disclaimer, and passive toxicity.

Figure 8: Correlating Perspective API scores with PaLM 2 safety feedback.

**arXiv Comp. Linguistics** @arxiv_cl@creative.ai · Aug 26, 2023

Aug 26, 2023

arXiv Comp. Linguistics @arxiv_cl@creative.ai

The Challenges of Machine Learning for Trust and Safety: A Case Study on Misinformation Detection

"Analyzes 270 well-cited papers and identify shortcomings in the literature that call into question claimed performance and practicality of automated misinformation detection models." [gal30b+] #LG #CL #CY

https://github.com/citp/sok_misinformation
https://arxiv.org/abs/2308.12215v1 #arxiv

Figure 3. Replication analysis of Nasir et al.: We edit the factual content of published news articles without significantly altering house style. Excerpted text from a Reuters news article is in the top panel; our edited version is in the bottom panel [330]. Text that differs between both versions is in color.

Figure 1. Corpus breakdown by year of publication and information scope. Note: A few papers belong to multiple information scopes.

Figure 2. Replication analysis of Ciampaglia et al.: Heatmap of moviedirector relationships, from a DBpedia dataset. Data is not binarized. Labeled points A, B, C, and D are false positive relationships.

**arXiv Comp. Linguistics** @arxiv_cl@creative.ai · Aug 23, 2023

Aug 23, 2023

arXiv Comp. Linguistics @arxiv_cl@creative.ai

LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

"LegalBench consists of 162 tasks covering 6 different types of legal reasoning, designed and hand-crafted by legal professionals, and evaluated against 20 open-source and commercial language models." [gal30b+] #CL #AI #CY

https://github.com/HazyResearch/legalbench/
https://arxiv.org/abs/2308.11462v1 #arxiv

Figure 1: We compare performance of prompts which describe the legal rule to be applied (“description”) against prompts which reference the legal rule to be applied (“reference”). Error bars measure standard error, computed using a bootstrap with 1000 resamples.

Figure 2: We compare performance of prompts which describe the task in plain language to prompts which describe the task in technical legal language (for GPT-3.5). Error bars measure standard error, computed using a bootstrap with 1000 resamples.

Figure 3: We evaluate GPT-3.5 and Incite-Instruct on five prompts constructed by randomly selecting different samples to use as in-context demonstrations (maintaining class balance in the prompt). In the figure above, each point corresponds to a different prompt.

Figure 4: LEGALBENCH task sizes and input text lengths.

**arXiv Comp. Linguistics** @arxiv_cl@creative.ai · Aug 2, 2023

Aug 2, 2023

arXiv Comp. Linguistics @arxiv_cl@creative.ai

Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias

"We examine three cognitive biases - the decoy effect, the certainty effect, and the belief bias - which are known to influence human decision-making and reasoning." [gal30b+] #CL #AI #CY #LG

https://github.com/google-research/
https://arxiv.org/abs/2308.00225v1 #arxiv

Recent searches

Search options

Administered by:

Server stats:

#cy