Just curious, how do you afford to automatically quant models?
It takes a lot of computing power, so I was curious.
It takes a lot of computing power, so I was curious.
@ozone-ai It indeed does cost a lot and we don’t earn anything for doing this. It is all voluntary and for the greater good for the AI community. We are donating our own time, hardware and hard-earned money for this project. We currently use 6 high-end servers for our operations which we constantly max out. 3 of them come from mradermacher, one from mradermachers boss, one from RichardErkhov and one from me. Richard’s server costs 250 Euros per month and is almost exclusively used to host rich1 for mradermacher quantization. I would expect similar cost for the other servers. Server donations are always welcome. In the end the cost of the time we invest into this project likely far exceeds any server cost as despite the process being automated a lot of manual work is required.
I myself am hosting nico1 at home. All the imatrix and mmproj extraction and a lot of quantization work happens there. It is our only server with enough RAM and the required GPUs to do imatrix computation. It uses an AMD Ryzen Threadripper PRO 7975WX as CPU, 512 GiB of RDIMM ECC DDR5 RAM and I reserve two RTX 4090 GPUs almost exclusively for mradermacher. I used to have relatively poor internet but upgraded to 10 Gbit fiber a few month ago to keep up with the around 250 TB of upload and 150 TB of download per month just for nico1. For me beside letting him use the hardware I own I mainly pay for electricity. We do our best to keep electricity cost as low as possible. On nico1 we run as much work as possible during daytime when I have free solar energy thanks to solar panels on my roof while only running urgent user requested models during the evening and running somewhat important models after 22:00 to make use of reduced nighttime electricity cost.
You can always check what our servers are doing under http://hf.tst.eu/status.html
That’s amazing. The amount of time, money, and effort you all put into this is truly dedication. I can tell you’re doing this purely for the love of the community and of AI. Thanks for breaking it all down and for everything you’re contributing!