Spaces:

KE-Team
/

KE-Omni

Running

App Files Files Community

shuaijiang commited on Nov 20, 2024

Commit

f040a31

1 Parent(s): 426adde

ke-omni

Browse files

Files changed (38) hide show

.gitattributes +3 -0
README.md +11 -10
index.html +261 -241
static/KE_Omni.pdf +3 -0
static/audios/en_162359_0_female.wav +3 -0
static/audios/en_162359_1_male.wav +3 -0
static/audios/en_21092_0_male.wav +3 -0
static/audios/en_21092_1_male.wav +3 -0
static/audios/en_52918_0_female.wav +3 -0
static/audios/en_52918_1_female.wav +3 -0
static/audios/en_7241_0_male.wav +3 -0
static/audios/en_7241_1_female.wav +3 -0
static/audios/exp_en_131152_0_mark.wav +3 -0
static/audios/exp_en_131152_1_mark.wav +3 -0
static/audios/exp_en_15479_0_mark.wav +3 -0
static/audios/exp_en_15479_1_mark.wav +3 -0
static/audios/exp_en_90.wav +3 -0
static/audios/exp_en_amomni_female_90.wav +3 -0
static/audios/exp_en_amomni_male_90.wav +3 -0
static/audios/exp_en_llamaomni_90.wav +3 -0
static/audios/exp_en_speechgpt_90.wav +3 -0
static/audios/zh_2518949_0_female.wav +3 -0
static/audios/zh_2518949_1_female.wav +3 -0
static/audios/zh_2615759_0_male.wav +3 -0
static/audios/zh_2615759_1_female.wav +3 -0
static/audios/zh_26157_0_male.wav +3 -0
static/audios/zh_26157_1_male.wav +3 -0
static/audios/zh_26178_0_female.wav +3 -0
static/audios/zh_26178_1_male.wav +3 -0
static/images/am_speech.png +3 -0
static/images/model.jpg +0 -0
static/images/speech.jpg +0 -0
static/images/speech.pdf +0 -0
static/images/speech.png +3 -0
static/images/subset.png +3 -0
static/images/textual.jpg +0 -0
static/images/textual.pdf +0 -0
static/images/textual.png +3 -0

.gitattributes CHANGED Viewed

@@ -46,3 +46,6 @@ static/videos/shiba.mp4 filter=lfs diff=lfs merge=lfs -text
 static/videos/steve.mp4 filter=lfs diff=lfs merge=lfs -text
 static/videos/teaser.mp4 filter=lfs diff=lfs merge=lfs -text
 static/videos/toby.mp4 filter=lfs diff=lfs merge=lfs -text

 static/videos/steve.mp4 filter=lfs diff=lfs merge=lfs -text
 static/videos/teaser.mp4 filter=lfs diff=lfs merge=lfs -text
 static/videos/toby.mp4 filter=lfs diff=lfs merge=lfs -text
+static/audios/*wav filter=lfs diff=lfs merge=lfs -text
+static/*pdf filter=lfs diff=lfs merge=lfs -text
+static/images/*png filter=lfs diff=lfs merge=lfs -text

README.md CHANGED Viewed

@@ -1,25 +1,26 @@
 ---
-title: KE Omni
 emoji: 🧠
 colorFrom: yellow
 colorTo: indigo
 sdk: static
 pinned: false
 ---
-# Nerfies
-This is the repository that contains source code for the [Nerfies website](https://nerfies.github.io).
-If you find Nerfies useful for your work please cite:
 ```
-@article{park2021nerfies
-  author    = {Park, Keunhong and Sinha, Utkarsh and Barron, Jonathan T. and Bouaziz, Sofien and Goldman, Dan B and Seitz, Steven M. and Martin-Brualla, Ricardo},
-  title     = {Nerfies: Deformable Neural Radiance Fields},
-  journal   = {ICCV},
-  year      = {2021},
 }
 ```
 # Website License
-<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.

 ---
+title: KE-Omni
 emoji: 🧠
 colorFrom: yellow
 colorTo: indigo
 sdk: static
 pinned: false
+license: apache-2.0
+short_description: KE-Omni
 ---
+# AM-Omni
+This is the repository that contains the description of AM-Omni.
+If you find AM-Omni useful for your work please cite:
 ```
+@article{zhao2024am,
+  author    = {Zhao, Shuajiang and Guo, Tingwei and Xiang, Bajian and Wan, Tongtang and Niu, Qiang and Zou, Wei and Li, Xiangang},
+  title     = {KE-Omni: A Seamless Speech Language Model with 60,000+ Hours Synthesized Chinese and English Speech Dialogues},
+  year      = {2024},
 }
 ```
 # Website License
+<a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>.

index.html CHANGED Viewed

@@ -3,10 +3,10 @@
 <head>
   <meta charset="utf-8">
   <meta name="description"
-        content="Deformable Neural Radiance Fields creates free-viewpoint portraits (nerfies) from casually captured videos.">
-  <meta name="keywords" content="Nerfies, D-NeRF, NeRF">
   <meta name="viewport" content="width=device-width, initial-scale=1">
-  <title>Nerfies: Deformable Neural Radiance Fields</title>
   <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
         rel="stylesheet">
@@ -33,39 +33,40 @@
     <div class="container is-max-desktop">
       <div class="columns is-centered">
         <div class="column has-text-centered">
-          <h1 class="title is-1 publication-title">Nerfies: Deformable Neural Radiance Fields</h1>
           <div class="is-size-5 publication-authors">
             <span class="author-block">
-              <a href="https://keunhong.com" target="_blank">Keunhong Park</a><sup>1</sup>,</span>
             <span class="author-block">
-              <a href="https://utkarshsinha.com" target="_blank">Utkarsh Sinha</a><sup>2</sup>,</span>
             <span class="author-block">
-              <a href="https://jonbarron.info" target="_blank">Jonathan T. Barron</a><sup>2</sup>,
             </span>
             <span class="author-block">
-              <a href="http://sofienbouaziz.com" target="_blank">Sofien Bouaziz</a><sup>2</sup>,
             </span>
             <span class="author-block">
-              <a href="https://www.danbgoldman.com" target="_blank">Dan B Goldman</a><sup>2</sup>,
             </span>
             <span class="author-block">
-              <a href="https://homes.cs.washington.edu/~seitz/" target="_blank">Steven M. Seitz</a><sup>1,2</sup>,
             </span>
             <span class="author-block">
-              <a href="http://www.ricardomartinbrualla.com" target="_blank">Ricardo Martin-Brualla</a><sup>2</sup>
             </span>
           </div>
           <div class="is-size-5 publication-authors">
-            <span class="author-block"><sup>1</sup>University of Washington,</span>
-            <span class="author-block"><sup>2</sup>Google Research</span>
           </div>
           <div class="column has-text-centered">
             <div class="publication-links">
               <!-- PDF Link. -->
               <span class="link-block">
-                <a href="https://arxiv.org/pdf/2011.12948" target="_blank"
                    class="external-link button is-normal is-rounded is-dark">
                   <span class="icon">
                       <i class="fas fa-file-pdf"></i>
@@ -74,7 +75,7 @@
                 </a>
               </span>
               <span class="link-block">
-                <a href="https://arxiv.org/abs/2011.12948" target="_blank"
                    class="external-link button is-normal is-rounded is-dark">
                   <span class="icon">
                       <i class="ai ai-arxiv"></i>
@@ -82,7 +83,7 @@
                   <span>arXiv</span>
                 </a>
               </span>
-              <!-- Video Link. -->
               <span class="link-block">
                 <a href="https://www.youtube.com/watch?v=MrKrnHhk8IA" target="_blank"
                    class="external-link button is-normal is-rounded is-dark">
@@ -92,9 +93,10 @@
                   <span>Video</span>
                 </a>
               </span>
               <!-- Code Link. -->
               <span class="link-block">
-                <a href="https://github.com/google/nerfies" target="_blank"
                    class="external-link button is-normal is-rounded is-dark">
                   <span class="icon">
                       <i class="fab fa-github"></i>
@@ -104,7 +106,7 @@
               </span>
               <!-- Dataset Link. -->
               <span class="link-block">
-                <a href="https://github.com/google/nerfies/releases/tag/0.1" target="_blank"
                    class="external-link button is-normal is-rounded is-dark">
                   <span class="icon">
                       <i class="far fa-images"></i>
@@ -112,7 +114,6 @@
                   <span>Data</span>
                   </a>
             </div>
           </div>
         </div>
       </div>
@@ -120,80 +121,6 @@
   </div>
 </section>
-<section class="hero teaser">
-  <div class="container is-max-desktop">
-    <div class="hero-body">
-      <video id="teaser" autoplay muted loop playsinline height="100%">
-        <source src="./static/videos/teaser.mp4"
-                type="video/mp4">
-      </video>
-      <h2 class="subtitle has-text-centered">
-        <span class="dnerf">Nerfies</span> turns selfie videos from your phone into
-        free-viewpoint
-        portraits.
-      </h2>
-    </div>
-  </div>
-</section>
-<section class="hero is-light is-small">
-  <div class="hero-body">
-    <div class="container">
-      <div id="results-carousel" class="carousel results-carousel">
-        <div class="item item-steve">
-          <video poster="" id="steve" autoplay controls muted loop playsinline height="100%">
-            <source src="./static/videos/steve.mp4"
-                    type="video/mp4">
-          </video>
-        </div>
-        <div class="item item-chair-tp">
-          <video poster="" id="chair-tp" autoplay controls muted loop playsinline height="100%">
-            <source src="./static/videos/chair-tp.mp4"
-                    type="video/mp4">
-          </video>
-        </div>
-        <div class="item item-shiba">
-          <video poster="" id="shiba" autoplay controls muted loop playsinline height="100%">
-            <source src="./static/videos/shiba.mp4"
-                    type="video/mp4">
-          </video>
-        </div>
-        <div class="item item-fullbody">
-          <video poster="" id="fullbody" autoplay controls muted loop playsinline height="100%">
-            <source src="./static/videos/fullbody.mp4"
-                    type="video/mp4">
-          </video>
-        </div>
-        <div class="item item-blueshirt">
-          <video poster="" id="blueshirt" autoplay controls muted loop playsinline height="100%">
-            <source src="./static/videos/blueshirt.mp4"
-                    type="video/mp4">
-          </video>
-        </div>
-        <div class="item item-mask">
-          <video poster="" id="mask" autoplay controls muted loop playsinline height="100%">
-            <source src="./static/videos/mask.mp4"
-                    type="video/mp4">
-          </video>
-        </div>
-        <div class="item item-coffee">
-          <video poster="" id="coffee" autoplay controls muted loop playsinline height="100%">
-            <source src="./static/videos/coffee.mp4"
-                    type="video/mp4">
-          </video>
-        </div>
-        <div class="item item-toby">
-          <video poster="" id="toby" autoplay controls muted loop playsinline height="100%">
-            <source src="./static/videos/toby2.mp4"
-                    type="video/mp4">
-          </video>
-        </div>
-      </div>
-    </div>
-  </div>
-</section>
 <section class="section">
   <div class="container is-max-desktop">
@@ -203,185 +130,279 @@
         <h2 class="title is-3">Abstract</h2>
         <div class="content has-text-justified">
           <p>
-            We present the first method capable of photorealistically reconstructing a non-rigidly
-            deforming scene using photos/videos captured casually from mobile phones.
           </p>
-          <p>
-            Our approach augments neural radiance fields
-            (NeRF) by optimizing an
-            additional continuous volumetric deformation field that warps each observed point into a
-            canonical 5D NeRF.
-            We observe that these NeRF-like deformation fields are prone to local minima, and
-            propose a coarse-to-fine optimization method for coordinate-based models that allows for
-            more robust optimization.
-            By adapting principles from geometry processing and physical simulation to NeRF-like
-            models, we propose an elastic regularization of the deformation field that further
-            improves robustness.
-          </p>
-          <p>
-            We show that <span class="dnerf">Nerfies</span> can turn casually captured selfie
-            photos/videos into deformable NeRF
-            models that allow for photorealistic renderings of the subject from arbitrary
-            viewpoints, which we dub <i>"nerfies"</i>. We evaluate our method by collecting data
-            using a
-            rig with two mobile phones that take time-synchronized photos, yielding train/validation
-            images of the same pose at different viewpoints. We show that our method faithfully
-            reconstructs non-rigidly deforming scenes and reproduces unseen views with high
-            fidelity.
-          </p>
-        </div>
-      </div>
-    </div>
-    <!--/ Abstract. -->
-    <!-- Paper video. -->
-    <div class="columns is-centered has-text-centered">
-      <div class="column is-four-fifths">
-        <h2 class="title is-3">Video</h2>
-        <div class="publication-video">
-          <iframe src="https://www.youtube.com/embed/MrKrnHhk8IA?rel=0&amp;showinfo=0"
-                  frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>
         </div>
       </div>
     </div>
-    <!--/ Paper video. -->
   </div>
 </section>
 <section class="section">
   <div class="container is-max-desktop">
     <div class="columns is-centered">
-      <!-- Visual Effects. -->
       <div class="column">
-        <div class="content">
-          <h2 class="title is-3">Visual Effects</h2>
-          <p>
-            Using <i>nerfies</i> you can create fun visual effects. This Dolly zoom effect
-            would be impossible without nerfies since it would require going through a wall.
-          </p>
-          <video id="dollyzoom" autoplay controls muted loop playsinline height="100%">
-            <source src="./static/videos/dollyzoom-stacked.mp4"
-                    type="video/mp4">
-          </video>
         </div>
       </div>
-      <!--/ Visual Effects. -->
-      <!-- Matting. -->
       <div class="column">
-        <h2 class="title is-3">Matting</h2>
-        <div class="columns is-centered">
-          <div class="column content">
-            <p>
-              As a byproduct of our method, we can also solve the matting problem by ignoring
-              samples that fall outside of a bounding box during rendering.
-            </p>
-            <video id="matting-video" controls playsinline height="100%">
-              <source src="./static/videos/matting.mp4"
-                      type="video/mp4">
-            </video>
-          </div>
         </div>
       </div>
     </div>
-    <!--/ Matting. -->
-    <!-- Animation. -->
     <div class="columns is-centered">
-      <div class="column is-full-width">
-        <h2 class="title is-3">Animation</h2>
-        <!-- Interpolating. -->
-        <h3 class="title is-4">Interpolating states</h3>
-        <div class="content has-text-justified">
-          <p>
-            We can also animate the scene by interpolating the deformation latent codes of two input
-            frames. Use the slider here to linearly interpolate between the left frame and the right
-            frame.
-          </p>
         </div>
-        <div class="columns is-vcentered interpolation-panel">
-          <div class="column is-3 has-text-centered">
-            <img src="./static/images/interpolate_start.jpg"
-                 class="interpolation-image"
-                 alt="Interpolate start reference image."/>
-            <p>Start Frame</p>
           </div>
-          <div class="column interpolation-video-column">
-            <div id="interpolation-image-wrapper">
-              Loading...
-            </div>
-            <input class="slider is-fullwidth is-large is-info"
-                   id="interpolation-slider"
-                   step="1" min="0" max="100" value="0" type="range">
           </div>
-          <div class="column is-3 has-text-centered">
-            <img src="./static/images/interpolate_end.jpg"
-                 class="interpolation-image"
-                 alt="Interpolation end reference image."/>
-            <p class="is-bold">End Frame</p>
           </div>
         </div>
-        <br/>
-        <!--/ Interpolating. -->
-        <!-- Re-rendering. -->
-        <h3 class="title is-4">Re-rendering the input video</h3>
-        <div class="content has-text-justified">
-          <p>
-            Using <span class="dnerf">Nerfies</span>, you can re-render a video from a novel
-            viewpoint such as a stabilized camera by playing back the training deformations.
-          </p>
         </div>
-        <div class="content has-text-centered">
-          <video id="replay-video"
-                 controls
-                 muted
-                 preload
-                 playsinline
-                 width="75%">
-            <source src="./static/videos/replay.mp4"
-                    type="video/mp4">
-          </video>
         </div>
-        <!--/ Re-rendering. -->
       </div>
     </div>
-    <!--/ Animation. -->
-    <!-- Concurrent Work. -->
     <div class="columns is-centered">
-      <div class="column is-full-width">
-        <h2 class="title is-3">Related Links</h2>
-        <div class="content has-text-justified">
-          <p>
-            There's a lot of excellent work that was introduced around the same time as ours.
-          </p>
-          <p>
-            <a href="https://arxiv.org/abs/2104.09125" target="_blank">Progressive Encoding for Neural Optimization</a> introduces an idea similar to our windowed position encoding for coarse-to-fine optimization.
-          </p>
-          <p>
-            <a href="https://www.albertpumarola.com/research/D-NeRF/index.html" target="_blank">D-NeRF</a> and <a href="https://gvv.mpi-inf.mpg.de/projects/nonrigid_nerf/" target="_blank">NR-NeRF</a>
-            both use deformation fields to model non-rigid scenes.
-          </p>
-          <p>
-            Some works model videos with a NeRF by directly modulating the density, such as <a href="https://video-nerf.github.io/" target="_blank">Video-NeRF</a>, <a href="https://www.cs.cornell.edu/~zl548/NSFF/" target="_blank">NSFF</a>, and <a href="https://neural-3d-video.github.io/" target="_blank">DyNeRF</a>
-          </p>
-          <p>
-            There are probably many more by the time you are reading this. Check out <a href="https://dellaert.github.io/NeRF/" target="_blank">Frank Dellart's survey on recent NeRF papers</a>, and <a href="https://github.com/yenchenlin/awesome-NeRF" target="_blank">Yen-Chen Lin's curated list of NeRF papers</a>.
-          </p>
         </div>
       </div>
     </div>
-    <!--/ Concurrent Work. -->
   </div>
 </section>
@@ -389,11 +410,10 @@
 <section class="section" id="BibTeX">
   <div class="container is-max-desktop content">
     <h2 class="title">BibTeX</h2>
-    <pre><code>@article{park2021nerfies,
-  author    = {Park, Keunhong and Sinha, Utkarsh and Barron, Jonathan T. and Bouaziz, Sofien and Goldman, Dan B and Seitz, Steven M. and Martin-Brualla, Ricardo},
-  title     = {Nerfies: Deformable Neural Radiance Fields},
-  journal   = {ICCV},
-  year      = {2021},
 }</code></pre>
   </div>
 </section>
@@ -403,10 +423,10 @@
   <div class="container">
     <div class="content has-text-centered">
       <a class="icon-link" target="_blank"
-         href="./static/videos/nerfies_paper.pdf">
         <i class="fas fa-file-pdf"></i>
       </a>
-      <a class="icon-link" href="https://github.com/keunhong" target="_blank" class="external-link" disabled>
         <i class="fab fa-github"></i>
       </a>
     </div>

 <head>
   <meta charset="utf-8">
   <meta name="description"
+        content="KE-Omni">
+  <meta name="keywords" content="speech language models, speech interaction">
   <meta name="viewport" content="width=device-width, initial-scale=1">
+  <title>KE-Omni</title>
   <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
         rel="stylesheet">
     <div class="container is-max-desktop">
       <div class="columns is-centered">
         <div class="column has-text-centered">
+          <h1 class="title is-1 publication-title">KE-Omni: A Seamless Speech Language Model with  60,000+ Hours Synthesized Chinese and English Speech Dialogues</h1>
           <div class="is-size-5 publication-authors">
             <span class="author-block">
+              Shuaijiang Zhao,
+            </span>
             <span class="author-block">
+               Tingwei Guo,
+            </span>
             <span class="author-block">
+               Bajian Xiang,
             </span>
             <span class="author-block">
+              Tongtang Wan,
             </span>
             <span class="author-block">
+              Qiang Niu,
             </span>
             <span class="author-block">
+              Wei Zou,
             </span>
             <span class="author-block">
+              Xiangang Li
             </span>
           </div>
           <div class="is-size-5 publication-authors">
+            <span class="author-block">Beike Inc., Beijing, China</span>
           </div>
           <div class="column has-text-centered">
             <div class="publication-links">
               <!-- PDF Link. -->
               <span class="link-block">
+                <a href="./static/KE_Omni.pdf" target="_blank"
                    class="external-link button is-normal is-rounded is-dark">
                   <span class="icon">
                       <i class="fas fa-file-pdf"></i>
                 </a>
               </span>
               <span class="link-block">
+                <a href="https://arxiv.org/" target="_blank"
                    class="external-link button is-normal is-rounded is-dark">
                   <span class="icon">
                       <i class="ai ai-arxiv"></i>
                   <span>arXiv</span>
                 </a>
               </span>
+              <!-- Video Link.
               <span class="link-block">
                 <a href="https://www.youtube.com/watch?v=MrKrnHhk8IA" target="_blank"
                    class="external-link button is-normal is-rounded is-dark">
                   <span>Video</span>
                 </a>
               </span>
+              -->
               <!-- Code Link. -->
               <span class="link-block">
+                <a href="https://github.com/LianjiaTech/BELLE" target="_blank"
                    class="external-link button is-normal is-rounded is-dark">
                   <span class="icon">
                       <i class="fab fa-github"></i>
               </span>
               <!-- Dataset Link. -->
               <span class="link-block">
+                <a href="https://huggingface.co/datasets/AM-Team/KESpeechChat" target="_blank"
                    class="external-link button is-normal is-rounded is-dark">
                   <span class="icon">
                       <i class="far fa-images"></i>
                   <span>Data</span>
                   </a>
             </div>
           </div>
         </div>
       </div>
   </div>
 </section>
 <section class="section">
   <div class="container is-max-desktop">
         <h2 class="title is-3">Abstract</h2>
         <div class="content has-text-justified">
           <p>
+            The GPT-4o represents a significant milestone in enabling real-time interaction with large language models (LLMs) through speech, its remarkable low latency and high fluency not only capture attention but also stimulate research interest in the field.
+            This real-time speech interaction is particularly valuable in scenarios requiring rapid feedback and immediate responses, dramatically enhancing user experience.
+            However, there is a notable lack of research focused on real-time large speech language models, particularly for Chinese.
+            In this work, we present KE-Omni, a seamless large speech language model built upon Ke-SpeechChat, a large-scale  high-quality synthesized  speech interaction dataset consisting of 7 million Chinese and English conversations, featuring 42,002 speakers, and totaling over 60,000 hours, This contributes significantly to the advancement of research and development in this field.
           </p>
         </div>
       </div>
     </div>
+    <!--/ Abstract. -->
   </div>
 </section>
 <section class="section">
   <div class="container is-max-desktop">
+    <h2 class="title is-2 has-text-centered" >KE-Omni Model Architecture</h2>
+    <p>
+      KE-Omni comprises three main components: a speech encoder, a large language model(LLM), and a speech decoder.
+    </p>
     <div class="columns is-centered">
       <div class="column">
+        <div class="content dialogue-block"  style="display: flex; flex-direction: column; justify-content: space-between; height: 100%;">
+            <img src="./static/images/model.jpg" alt="am-omni architecture">
         </div>
       </div>
+    </div>
+    <h2 class="title is-2 has-text-centered" >KE-SpeechChat Dataset Construction</h2>
+    <div class="columns is-centered">
       <div class="column">
+        <div class="content dialogue-block"  style="display: flex; flex-direction: column; justify-content: space-between; height: 100%;">
+          <h3 class="title is-4">Textual Dialogue Construction</h3>
+            <img src="./static/images/textual.png" alt="textual dialogue construction">
+          <h3 class="title is-4">Speech Dialogue Construction</h3>
+            <img src="./static/images/speech.png" alt="speech dialogue construction">
         </div>
       </div>
     </div>
+    <h2 class="title is-2 has-text-centered">KE-SpeechChat Dataset Details</h2>
     <div class="columns is-centered">
+      <!-- Left image -->
+      <div class="column is-half">
+        <div class="content dialogue-block" style="display: flex; flex-direction: column; justify-content: space-between; height: 100%;">
+          <h3 class="title is-4 has-text-centered">Statistics of KESpeechChat</h3>
+          <img src="./static/images/am_speech.png" alt="Statistics of KESpeechChat " style="width: 100%; height: 100%;">
+        </div>
+      </div>
+      <!-- Right image -->
+      <div class="column is-half">
+        <div class="content dialogue-block" style="display: flex; flex-direction: column; justify-content: space-between; height: 100%;">
+          <h3 class="title is-4 has-text-centered">Subsets of KESpeechChat</h3>
+          <img src="./static/images/subset.png" alt="Subsets of KESpeechChat" style="width: 100%; height: 100%;">
         </div>
+      </div>
+    </div>
+  </div>
+</section>
+<section class="section">
+  <div class="container is-max-desktop">
+    <h2 class="title is-2 has-text-centered" >KESpeechChat Dialogue Demos</h2>
+    <h2 class="title is-3 has-text-centered" >Chinese Dialogues</h2>
+    <div class="columns is-centered">
+      <div class="column">
+        <div class="content dialogue-block"  style="display: flex; flex-direction: column; justify-content: space-between; height: 100%;">
+          <h3 class="title is-4">User-male, Agent-male</h3>
+          <p>User: 奥运会是不是每四年举办一次啊</p>
+          <p>Agent: 是的，奥运会确实是每四年举办一次，不过有时候会因为特殊情况有所调整，比如之前就因为全球健康危机延期过。</p>
+          <div style="margin-top: auto;">
+            <audio controls>
+              <source src="./static/audios/zh_26157_0_male.wav" type="audio/wav">
+            </audio>
+            <audio controls>
+              <source src="./static/audios/zh_26157_1_male.wav" type="audio/wav">
+            </audio>
           </div>
+        </div>
+      </div>
+      <div class="column">
+        <div class="content dialogue-block"  style="display: flex; flex-direction: column; justify-content: space-between; height: 100%;">
+          <h3 class="title is-4">User-male, Agent-female</h3>
+          <p>User: 人工智能现在都用在哪些地方了？</p>
+          <p>Agent: 人工智能的应用范围非常广泛，比如智能家居、自动驾驶汽车、医疗诊断、金融分析、教育辅导、娱乐推荐系统等等，几乎涵盖了我们生活的方方面面。</p>
+          <div style="margin-top: auto;">
+            <audio controls>
+              <source src="./static/audios/zh_2615759_0_male.wav" type="audio/wav">
+            </audio>
+            <audio controls>
+              <source src="./static/audios/zh_2615759_1_female.wav" type="audio/wav">
+            </audio>
           </div>
+        </div>
+      </div>
+      <div class="column">
+        <div class="content"  style="display: flex; flex-direction: column; justify-content: space-between; height: 100%;">
+          <h3 class="title is-4">User-female, Agent-male</h3>
+          <p>User: 人类的平均体温一般是多少度呀？</p>
+          <p>Agent: 人类的平均体温通常是在三十六点五到三十七点五摄氏度之间，这个范围内的体温都算是正常的。</p>
+          <div style="margin-top: auto;">
+            <audio controls>
+              <source src="./static/audios/zh_26178_0_female.wav" type="audio/wav">
+            </audio>
+            <audio controls>
+              <source src="./static/audios/zh_26178_1_male.wav" type="audio/wav">
+            </audio>
+          </div>
+        </div>
+      </div>
+      <div class="column">
+        <div class="content"  style="display: flex; flex-direction: column; justify-content: space-between; height: 100%;">
+          <h3 class="title is-4">User-female, Agent-female</h3>
+          <p>User: 如果我想提高写作水平，多读书真的有帮助吗？</p>
+          <p>Agent: 当然有帮助，多读书可以让你接触到不同的写作风格和表达方式，还能增加词汇量，对提高写作水平特别有用。</p>
+          <div style="margin-top: auto;">
+            <audio controls>
+              <source src="./static/audios/zh_2518949_0_female.wav" type="audio/wav">
+            </audio>
+            <audio controls>
+              <source src="./static/audios/zh_2518949_1_female.wav" type="audio/wav">
+            </audio>
           </div>
         </div>
+      </div>
+    </div>
+  </div>
+</section>
+<section class="section">
+  <div class="container is-max-desktop">
+    <h2 class="title is-3 has-text-centered">English Dialogues</h2>
+    <div class="columns is-centered">
+      <div class="column">
+        <div class="content dialogue-block"  style="display: flex; flex-direction: column; justify-content: space-between; height: 100%;">
+          <h3 class="title is-4">User-male, Agent-male</h3>
+          <p>User: What kinds of birds and trees would you usually see in a peaceful forest clearing?</p>
+          <p>Agent: In a peaceful forest clearing, you might see a variety of birds like robins, sparrows, and woodpeckers. As for trees, you could expect to find oaks, maples, and pines, which provide a beautiful and serene backdrop.</p>
+          <div style="margin-top: auto;">
+            <audio controls>
+              <source src="./static/audios/en_21092_0_male.wav" type="audio/wav">
+            </audio>
+            <audio controls>
+              <source src="./static/audios/en_21092_1_male.wav" type="audio/wav">
+            </audio>
+          </div>
         </div>
+      </div>
+      <div class="column">
+        <div class="content dialogue-block"  style="display: flex; flex-direction: column; justify-content: space-between; height: 100%;">
+          <h3 class="title is-4">User-male, Agent-female</h3>
+          <p>User: What are some things I should think about before agreeing to speak at a conference?</p>
+          <p>Agent: When considering a speaking engagement, think about the alignment of the conference theme with your expertise, the audience demographics, the potential networking opportunities, the time commitment required, and any travel or financial arrangements. Also, consider how this opportunity fits into your career goals and whether it can help you reach a wider audience or gain new insights.</p>
+          <div style="margin-top: auto;">
+            <audio controls>
+              <source src="./static/audios/en_7241_0_male.wav" type="audio/wav">
+            </audio>
+            <audio controls>
+              <source src="./static/audios/en_7241_1_female.wav" type="audio/wav">
+            </audio>
+          </div>
+        </div>
+      </div>
+      <div class="column">
+        <div class="content"  style="display: flex; flex-direction: column; justify-content: space-between; height: 100%;">
+          <h3 class="title is-4">User-female, Agent-male</h3>
+          <p>User: Could you tell me what the capital of France is?</p>
+          <p>Agent: The capital of France is Paris.</p>
+          <div style="margin-top: auto;">
+            <audio controls>
+              <source src="./static/audios/en_162359_0_female.wav" type="audio/wav">
+            </audio>
+            <audio controls>
+              <source src="./static/audios/en_162359_1_male.wav" type="audio/wav">
+            </audio>
+          </div>
+        </div>
+      </div>
+      <div class="column">
+        <div class="content"  style="display: flex; flex-direction: column; justify-content: space-between; height: 100%;">
+          <h3 class="title is-4">User-female, Agent-female</h3>
+          <p>User: What's the best way to store different types of garden produce to keep them fresh for as long as possible?</p>
+          <p>Agent: To keep your garden produce fresh, it's best to store each type according to its specific needs. For example, leafy greens like lettuce and spinach should be kept in a cool, humid environment, while root vegetables like carrots and potatoes prefer a cooler, drier place. Berries and stone fruits do well in the refrigerator, but tomatoes should be stored at room temperature. Make sure to avoid storing ethylene-producing fruits like apples and bananas with sensitive vegetables, as this can cause premature ripening.</p>
+          <div style="margin-top: auto;">
+            <audio controls>
+              <source src="./static/audios/en_52918_0_female.wav" type="audio/wav">
+            </audio>
+            <audio controls>
+              <source src="./static/audios/en_52918_1_female.wav" type="audio/wav">
+            </audio>
+          </div>
         </div>
       </div>
     </div>
+  </div>
+</section>
+<section class="section">
+  <div class="container is-max-desktop">
+    <h2 class="title is-2 has-text-centered">Experiment Results</h2>
+    <h2 class="title is-3 has-text-centered">English Dialogues</h2>
     <div class="columns is-centered">
+      <div class="column">
+        <div class="content dialogue-block"  style="display: flex; flex-direction: column; justify-content: space-between; height: 100%;">
+          <h3 class="title is-4">Ground Truth</h3>
+          <p>User: What kind of job would suit someone who likes solving problems and interacting with people?</p>
+          <p>Agent: A career in fields like customer service, human resources, or project management could be a great fit. These roles often require strong problem-solving skills and regular interaction with people, which can be both challenging and rewarding.</p>
+          <div style="margin-top: auto;">
+            <audio controls>
+              <source src="./static/audios/exp_en_131152_0_mark.wav" type="audio/wav">
+            </audio>
+            <audio controls>
+              <source src="./static/audios/exp_en_131152_1_mark.wav" type="audio/wav">
+            </audio>
+          </div>
+        </div>
+      </div>
+      <div class="column">
+        <div class="content dialogue-block"  style="display: flex; flex-direction: column; justify-content: space-between; height: 100%;">
+          <h3 class="title is-4">SpeechGPT</h3>
+          <p>User: What kind of job would suit someone who likes solving problems and interacting with people?</p>
+          <p>Agent: You may be referring to a job in customer service. This type of job involves interacting with customers and solving their problems. It requires excellent communication skills and the ability to remain calm under pressure.</p>
+          <div style="margin-top: auto;">
+            <audio controls>
+              <source src="./static/audios/exp_en_131152_0_mark.wav" type="audio/wav">
+            </audio>
+            <audio controls>
+              <source src="./static/audios/exp_en_speechgpt_90.wav" type="audio/wav">
+            </audio>
+          </div>
+        </div>
+      </div>
+      <div class="column">
+        <div class="content"  style="display: flex; flex-direction: column; justify-content: space-between; height: 100%;">
+          <h3 class="title is-4">LLaMA-Omni</h3>
+          <p>User: What kind of job would suit someone who likes solving problems and interacting with people?</p>
+          <p>Agent: A job that suits someone who likes solving problems and interacting with people is a consultant or a project manager.</p>
+          <div style="margin-top: auto;">
+            <audio controls>
+              <source src="./static/audios/exp_en_131152_0_mark.wav" type="audio/wav">
+            </audio>
+            <audio controls>
+              <source src="./static/audios/exp_en_llamaomni_90.wav" type="audio/wav">
+            </audio>
+          </div>
+        </div>
+      </div>
+      <div class="column">
+        <div class="content"  style="display: flex; flex-direction: column; justify-content: space-between; height: 100%;">
+          <h3 class="title is-4">KE-Omni</h3>
+          <p>User: What kind of job would suit someone who likes solving problems and interacting with people?</p>
+          <p>Agent: A job that combines problem-solving skills with people interaction could be a customer service representative or a project manager. These roles often require creative thinking to resolve conflicts and communicate effectively with various clients and team members. Another great option might be a salesperson, where you can build strong relationships and help clients find the right solutions.</p>
+          <div style="margin-top: auto;">
+            <audio controls>
+              <source src="./static/audios/exp_en_131152_0_mark.wav" type="audio/wav">
+            </audio>
+            <audio controls>
+              <source src="./static/audios/exp_en_amomni_female_90.wav" type="audio/wav">
+            </audio>
+          </div>
         </div>
       </div>
     </div>
   </div>
 </section>
 <section class="section" id="BibTeX">
   <div class="container is-max-desktop content">
     <h2 class="title">BibTeX</h2>
+    <pre><code>@article{zhao2024keomni,
+  author    = {Zhao, Shuajiang and Guo, Tingwei and Xiang, Bajian and Wan, Tongtang and Niu, Qiang and Zou, Wei and Li, Xiangang},
+  title     = {KE-Omni: A Seamless Speech Language Model with 60,000+ Hours Synthesized Chinese and English Speech Dialogues},
+  year      = {2024},
 }</code></pre>
   </div>
 </section>
   <div class="container">
     <div class="content has-text-centered">
       <a class="icon-link" target="_blank"
+         href="./static/am_omni.pdf">
         <i class="fas fa-file-pdf"></i>
       </a>
+      <a class="icon-link" href="https://github.com/LianjiaTech/BELLE" target="_blank" class="external-link" disabled>
         <i class="fab fa-github"></i>
       </a>
     </div>

static/KE_Omni.pdf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e54bf3b22998263825af29eefb0d16ebe9db3d818abb7723c5fc47a9bbcbdb8b
+size 1824220

static/audios/en_162359_0_female.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:575823556925c1e8b72d3271586a268a816302ff87711ddc20e1f5e3f999fff8
+size 257930

static/audios/en_162359_1_male.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dccd4e7382403e9c64e15b13c601cd8b06105bc84a14016887f68e9b7d1dae09
+size 160650

static/audios/en_21092_0_male.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:df12f9421f723ce5f888728f24bcb3f25e2af77ebb98b544c32dfd5ce02f2e60
+size 532362

static/audios/en_21092_1_male.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:70a9475dad1c5daa4c5823a3971138803abaf0b0fd8c444fb01f4bce86ce0712
+size 1451914

static/audios/en_52918_0_female.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:859760f911b518295dba512f14e55aefa90c5983a4f55411103c8938e4a8d200
+size 539530

static/audios/en_52918_1_female.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:64f3bb84808f1098976f2e82fa00b0605eb406133dea2d1ef99de3f5a799f45a
+size 2726630

static/audios/en_7241_0_male.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ce4f307dc7fa53808a4c12d56d23b6f89b40e330a3272172ba972642472e65ac
+size 379786

static/audios/en_7241_1_female.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7e6b9865ca8d1dfc7c33679f9921cee3579a04117d52a3fdfa32014495dbbadb
+size 2069386

static/audios/exp_en_131152_0_mark.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5ff873833fefe83390399efa6e1482c21d2978b952850f1f89dd6024fe9b85f7
+size 495498

static/audios/exp_en_131152_1_mark.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e451a1423a8ffed1e86286b64840c104c2ffdcd9f80190ff2e6da9a03a538693
+size 1371018

static/audios/exp_en_15479_0_mark.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7caa8384db06ab9ff1ca51930bbd5946e0acbcc2150a819b0eccf1b533596c09
+size 626570

static/audios/exp_en_15479_1_mark.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c618df62c94b5dea7889679355d4f8428045286b62c1f4e42b181085f46b101a
+size 2147210

static/audios/exp_en_90.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:48d5c875b4f36bfbb9ab7c3be4704a8da8f18de96711fc7e80bbe0f959899348
+size 291244

static/audios/exp_en_amomni_female_90.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:23c8dd75b89815c6a3c802eb3fab649fb9873480e930e0c32b165f3026f5f84d
+size 518444

static/audios/exp_en_amomni_male_90.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:48d5c875b4f36bfbb9ab7c3be4704a8da8f18de96711fc7e80bbe0f959899348
+size 291244

static/audios/exp_en_llamaomni_90.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c25cc2cd6011b97bbb97dfafb62f7b910e21865861d921b2d3db0772f237548c
+size 212524

static/audios/exp_en_speechgpt_90.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1d2cd8d2833ceb7d50c9f2807794edbefc333c171a938fb28656e82a60d30038
+size 407724

static/audios/zh_2518949_0_female.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a9355c80c26bafbead3abe5cb4201d95aef0b4572913c91c02107ffa14d402e1
+size 462730

static/audios/zh_2518949_1_female.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a47318dfdc021492a0630dc893be8b7748ea642387301225070ef7f9fee76878
+size 946058

static/audios/zh_2615759_0_male.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:91c737e5767879c4f539ef5b750f3f595c28e069bcc0c17174e29e9254f8d48f
+size 375690

static/audios/zh_2615759_1_female.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ae0f5af806a9628f99bb8f964589d79964fa5147ccec523c35e6862c32f0a8c7
+size 1183626

static/audios/zh_26157_0_male.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b1beb2ed3f0bf3711615be8271be82ecee5919814dd2e79eef54cf1db0ed055a
+size 341898

static/audios/zh_26157_1_male.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bae4b72b41e6fe9f42ef3dd4071cf53ca28bc717773dbdb66d4fde8082c5ff96
+size 907146

static/audios/zh_26178_0_female.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2e869fa9b2052a7d3a61554975ab6baad7c8e77080c59350ee3e3353e5176f65
+size 354186

static/audios/zh_26178_1_male.wav ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ac98f8baaafeb171f49ef3e715ced8d41144a42c665f1a6bc99a45a168a6a11a
+size 855946

static/images/am_speech.png ADDED Viewed

Git LFS Details

SHA256: ad7d6f09cc77afe84403672eaaed0ad7f9a68fff083bda2344c26415243f4af4
Pointer size: 130 Bytes
Size of remote file: 67.8 kB

static/images/model.jpg ADDED Viewed

static/images/speech.jpg ADDED Viewed

static/images/speech.pdf ADDED Viewed

Binary file (714 kB). View file

static/images/speech.png ADDED Viewed

Git LFS Details

SHA256: 218b6f488cb25264198ae40833a07d501696dca4f121330e00419522a13c9913
Pointer size: 132 Bytes
Size of remote file: 1.05 MB

static/images/subset.png ADDED Viewed

Git LFS Details

SHA256: 4e958e802a6ddde79923993cfce9af288c7174c7546481e9483e09d1a42e80d4
Pointer size: 130 Bytes
Size of remote file: 57.2 kB

static/images/textual.jpg ADDED Viewed

static/images/textual.pdf ADDED Viewed

Binary file (457 kB). View file

static/images/textual.png ADDED Viewed

Git LFS Details

SHA256: 0084d37bf73bdbf16a5d8b2e2b25c5156ac0b486df511c1f45d5ce32a827a508
Pointer size: 132 Bytes
Size of remote file: 1.22 MB