aminabbasi/PsychoLexLLaMA-8B

PsychoLexLLaMA: A Specialized LLM for Psychology

PsychoLexLLaMA is a specialized large language model (LLM) meticulously designed to enhance the understanding and application of psychological concepts. As part of the research paper titled "PsychoLex: Unveiling the Psychological Mind of Large Language Models", this model has been optimized for efficiency, requiring minimal data and hardware resources while surpassing the performance of general-purpose LLMs in psychological contexts.

Model Overview

PsychoLexLLaMA is built upon the LLaMA 3.1 architecture, with a focus on psychological content. The model is available in two configurations: 8B and 70B parameters. It has undergone a multi-phase development process, including continuous pre-training, supervised fine-tuning, and a linear weight combination strategy, ensuring a robust and specialized model for psychological research and applications.

Here's the code for using the PsychoLexLLaMA:

import transformers
import torch

# Specify the model ID for PsychoLexLLaMA
model_id = "aminabbasi/PsychoLexLLaMA-8B"

# Initialize the text generation pipeline with the model
pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

# Define a sample input, with a system prompt for context and a user prompt for a query
messages = [
    {"role": "system", "content": "You are a psychology expert chatbot designed to assist with psychological questions."},
    {"role": "user", "content": "Can you explain cognitive dissonance?"},
]

# Generate output from the model
outputs = pipeline(
    messages,
    max_new_tokens=256,
)

# Print the generated response
print(outputs[0]["generated_text"])

1. Continuous Pre-Training

The first phase of development involved continuous pre-training using the LoRA technique on the bilingual texts of "Introduction to Psychology" by Hilgard, processed in both Persian and English. This foundational training aligned the base models with psychological content, enhancing their ability to understand and apply complex psychological concepts efficiently.

• Base Models: LLaMA 3.1 (8B)
• Training Technique: LoRA (Low-Rank Adaptation)
• Languages: Persian and English

2. Supervised Fine-Tuning

In the supervised fine-tuning phase, PsychoLexLLaMA was further refined using the PsychoLexQA dataset, which includes detailed instructional content and a comprehensive set of questions and answers. This phase was crucial in honing the model’s ability to accurately interpret and respond to intricate psychological queries and scenarios.

• Dataset: PsychoLexQA
• Purpose: Enhance model’s performance in psychological analysis

3. Linear Weight Combination

To ensure robustness and preserve the integrity of the previous training, a linear weight combination strategy was employed. This involved merging the weights of the LLaMA 3.1 Instruct model with those of our continuously pre-trained and fine-tuned models. Each model contributed 50% of its weight to create a balanced and potent tool, adept at handling sophisticated psychological inquiries.

• Weight Contribution: 50% LLaMA 3.1 Instruct, 50% PsychoLexLLaMA
• Result: A model with enhanced psychological expertise

Key Features

• Psychology-Specific Tuning: Optimized specifically for psychological applications, making it highly effective in understanding and generating content related to psychological studies.
• Bilingual Capability: Trained on both Persian and English texts, enabling robust performance in multilingual contexts.
• Efficient Resource Usage: Designed to perform effectively with minimal data and hardware requirements, making it accessible for a wider range of research applications.
• Robust Performance: Combines foundational LLaMA capabilities with specialized psychological training, resulting in a model that excels in complex psychological tasks.

Applications

PsychoLexLLaMA is ideal for:

• Research in Psychology: Enhance research methodologies with a model specifically tuned for psychological content.
• AI-Driven Psychological Tools: Develop applications and tools that require a deep understanding of psychological concepts.
• Educational Resources: Create educational materials that leverage the model's expertise in psychology.

Citation

If you use PsychoLexLLaMA in your research, please cite our paper:

@article{Abbasi2024PsychoLex,
  title={PsychoLex: Unveiling the Psychological Mind of Large Language Models},
  author={Mohammad Amin Abbasi and Farnaz Sadat Mirnezami and Hassan Naderi},
}

License

This model is available under the CC BY-NC 4.0 License. This license permits non-commercial use with proper attribution, allowing others to build upon the work under the same terms.

Please ensure to adhere to these license terms when using the model.