---
datasets:
- zetavg/ShareGPT-Processed
- zetavg/coct-en-zh-tw-translations-twp-300k
- zetavg/zh-tw-wikipedia
- zetavg/tw-sinica-corpus-word-frequency
- RyokoAI/ShareGPT52K
language:
- zh
- en
---
# TW-Pythia-6.9B-Chat

**Taiwanese Mandarin Pythia Language Model, instruction-tuned for dialogue.**

Version 0.2

## Model Details

The TW-Pythia model is derived from the Apache-2.0-licenced [Pythia](https://github.com/EleutherAI/pythia) language model, with 8000 new Traditional Chinese tokens added, embed layers resized and re-trained.

### Basics

- **Developed by:** [@zetavg](https://github.com/zetavg) based on [EleutherAI](https://www.eleuther.ai/)'s [Pythia](https://github.com/EleutherAI/pythia) language model.
- **Model type:** Transformer-based GPT-NeoX Causal Language Model
- **Languages:** English, Traditional Chinese
- **License:** Unknown due to unconfirmed usage license of the training data
- **Derived from model:** [EleutherAI/pythia-6.9b](https://huggingface.co./EleutherAI/pythia-6.9b)

### Model Sources

- **Repository:** https://github.com/zetavg/twlm
- **Demo:** See https://hackmd.io/@z/twlm-demo

## Uses

Currently, this model has not demonstrated any practical value in Traditional Chinese processing without further training, but it does possess some basic Chinese-English translation capabilities.

## Training Details

### Training Data

* 200k [English ↔ Traditional Chinese Sentences from the COCT Database](zetavg/coct-en-zh-tw-translations-twp-300k).
* ~8k English and Traditional Chinese mixed [ShareGPT data](zetavg/ShareGPT-Processed).

### Training Procedure

First, we build a BPE tokenizer based on the original Pythia tokenizer with 8000 new Traditional Chinese tokens added.

Then, we resize the embedding layer of the `pythia-6.9b` model to accommodate the new vocabulary size, and we train only the input/output embedding layers to allow the model to learn the new Traditional Chinese words and phrases.

At last, LoRA weights are added to the model and fine-tuned for instruction following.

#### Training Hyperparameters

- **Training regime:** `fp32`
- See: https://github.com/zetavg/twlm/blob/main/configs/ta01_p7b.yaml

### Hardware

* 1xH100 80GB GPU on Lambda Cloud (with Skypilot), about 20h in total.