heartmula

Generate music from lyrics and tags using PyTorch on CUDA-enabled GPUs or CPUs.

27|2|Updated Jan 15, 2024
One-click install
npx skills add https://github.com/erfanzar/Xerxes-Agents --skill heartmula-erfanzar
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: heartmula
Source: https://github.com/erfanzar/Xerxes-Agents/tree/main/src/python/xerxes/skills/media/heartmula
Command: npx skills add https://github.com/erfanzar/Xerxes-Agents --skill heartmula-erfanzar

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

HeartMuLa provides an open-source, offline music generation workflow that creates songs from lyrics and tags while keeping data local and private.

Core Features & Use Cases

  • HeartMuLa: a music language model (3B/7B) that generates music from lyrics and tags
  • HeartCodec: a 12.5Hz codec for high-fidelity audio reconstruction
  • HeartTranscriptor: Whisper-based lyrics transcription
  • HeartCLAP: audio-text alignment
  • Hardware-friendly: lazy loading and multi-GPU guidance for local setups

Quick Start

Run a local generation workflow by supplying lyrics and tags to HeartMuLa on a CUDA-enabled GPU and save the output to an audio file.

Frequently Asked Questions about heartmula

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate music from lyrics locally without uploading my data?

To generate music from lyrics locally, you can use an open-source model like HeartMuLa that processes text and tags entirely offline on your own hardware, ensuring your data remains completely private and secure.

Do I need a CUDA-enabled GPU to run offline music generation?

While a CUDA-enabled GPU is recommended for offline music generation to handle the model pipeline efficiently, you can also run the process on a CPU, though generation times will be significantly slower.

What is needed to set up a local environment for open-source music generation?

Setting up local open-source music generation requires a Python environment, PyTorch, and downloadable HeartMuLa checkpoints to execute the generation pipeline and save the output to an audio file.

Can I use machine learning to transcribe audio and align text for music generation?

Yes, this workflow includes HeartTranscriptor for Whisper-based lyrics transcription and HeartCLAP for audio-text alignment, enabling accurate machine learning audio processing and high-fidelity reconstruction.

How does high-fidelity audio reconstruction work for offline generated music?

High-fidelity audio reconstruction for offline generated music is handled by HeartCodec, a 12.5Hz codec that accurately translates the generated model outputs back into clear, high-quality audio files.

Does multi-GPU guidance support rapid prototyping for music generation?

Yes, the pipeline features lazy loading and multi-GPU guidance to optimize hardware resources, making it highly suitable for rapid prototyping and experimental sound design on local setups.